2024-01-16T13:40:58.092 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T13:40:58.092 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127 --name rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi --verbose --owner scheduled_rishabh@teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} -- /home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127/orig.config.yaml 2024-01-16T13:40:58.134 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T13:40:58.210 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7518127' kernel: client: sha1: distro flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1988 sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMgWGcqZsqMrhLUrN+rzIKJxjgom5ezT+W4YHAhRRit0V/oXcrcFOoaNVAfVwWQB1r4CmW4qRPPv3PnYCrSBGEE= smithi124.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGv3mk9zPlpUpN698e9yQhF8X8unVOZl5D0AVavTlbH00i8d2sN7KnsA82RC5i+0EbqLMIENWV7ImBDF2RWbi4Q= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-16_13:34:04 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T13:40:58.210 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-16T13:40:58.210 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-16T13:40:58.210 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T13:40:58.212 INFO:teuthology.task.internal:Checking packages... 2024-01-16T13:40:58.232 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '73a4532fc4d97adee0b985d4d42874ea351a1c86' 2024-01-16T13:40:58.232 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T13:40:58.232 INFO:teuthology.packaging:ref: None 2024-01-16T13:40:58.232 INFO:teuthology.packaging:tag: None 2024-01-16T13:40:58.232 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-16T13:40:58.232 INFO:teuthology.packaging:sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T13:40:58.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-clone-stats 2024-01-16T13:40:58.450 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g73a4532f 2024-01-16T13:40:58.451 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T13:40:58.458 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T13:40:58.458 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T13:40:58.506 INFO:teuthology.task.internal:Saving configuration 2024-01-16T13:40:58.521 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T13:40:58.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T13:40:58.554 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 13:35:19.388062', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMgWGcqZsqMrhLUrN+rzIKJxjgom5ezT+W4YHAhRRit0V/oXcrcFOoaNVAfVwWQB1r4CmW4qRPPv3PnYCrSBGEE='} 2024-01-16T13:40:58.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 13:35:19.390188', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGv3mk9zPlpUpN698e9yQhF8X8unVOZl5D0AVavTlbH00i8d2sN7KnsA82RC5i+0EbqLMIENWV7ImBDF2RWbi4Q='} 2024-01-16T13:40:58.571 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T13:40:58.578 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-16T13:40:58.578 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-16T13:40:58.578 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T13:40:58.666 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4f4ec18160>, signals=[15]) 2024-01-16T13:40:58.666 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T13:40:58.671 INFO:teuthology.task.internal:Opening connections... 2024-01-16T13:40:58.671 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:40:58.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T13:40:58.745 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:40:58.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T13:40:58.817 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T13:40:58.823 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-01-16T13:40:58.844 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-01-16T13:40:58.844 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-01-16T13:40:58.903 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="Red Hat Enterprise Linux" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="8.6 (Ootpa)" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:ID="rhel" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="fedora" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="8.6" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el8" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://www.redhat.com/" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-16T13:40:58.904 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-16T13:40:58.905 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-01-16T13:40:58.927 DEBUG:teuthology.orchestra.run.smithi124:> uname -m 2024-01-16T13:40:58.947 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2024-01-16T13:40:58.947 DEBUG:teuthology.orchestra.run.smithi124:> cat /etc/os-release 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="Red Hat Enterprise Linux" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="8.6 (Ootpa)" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:ID="rhel" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="fedora" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="8.6" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el8" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://www.redhat.com/" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:40:59.006 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T13:40:59.007 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-16T13:40:59.007 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-16T13:40:59.007 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-16T13:40:59.007 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2024-01-16T13:40:59.027 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T13:40:59.036 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T13:40:59.056 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T13:40:59.057 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-01-16T13:40:59.059 DEBUG:teuthology.orchestra.run.smithi124:> test '!' -e /home/ubuntu/cephtest 2024-01-16T13:40:59.076 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T13:40:59.082 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T13:40:59.082 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-01-16T13:40:59.117 DEBUG:teuthology.orchestra.run.smithi124:> test -z $(ls -A /var/lib/ceph) 2024-01-16T13:40:59.141 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T13:40:59.191 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T13:40:59.204 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-16T13:40:59.204 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-01-16T13:40:59.204 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-01-16T13:40:59.205 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T13:40:59.205 DEBUG:teuthology.orchestra.run.smithi124:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T13:40:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:40:59.222 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2024-01-16T13:40:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:40:59.224 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-01-16T13:40:59.281 INFO:teuthology.orchestra.run.smithi124.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:40:59.281 INFO:teuthology.task.kernel:Running kernel on smithi124: 4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:40:59.281 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum install -y kernel 2024-01-16T13:40:59.283 INFO:teuthology.orchestra.run.smithi113.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:40:59.283 INFO:teuthology.task.kernel:Running kernel on smithi113: 4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:40:59.284 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum install -y kernel 2024-01-16T13:41:00.742 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2024-01-16T13:41:00.742 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2024-01-16T13:41:00.821 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-16T13:41:00.821 INFO:teuthology.orchestra.run.smithi113.stdout:Unable to read consumer identity 2024-01-16T13:41:00.843 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:41:00.843 INFO:teuthology.orchestra.run.smithi124.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-16T13:41:00.843 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:41:00.954 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:41:00.954 INFO:teuthology.orchestra.run.smithi113.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-16T13:41:00.954 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:41:02.681 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-16T13:41:02.707 INFO:teuthology.orchestra.run.smithi124.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-16T13:41:03.558 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2024-01-16T13:41:04.482 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2024-01-16T13:41:07.183 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 331 kB/s | 24 kB 00:00 2024-01-16T13:41:07.952 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-16T13:41:07.953 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-16T13:41:08.037 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2024-01-16T13:41:08.039 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2024-01-16T13:41:08.039 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2024-01-16T13:41:08.140 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 374 kB/s | 24 kB 00:00 2024-01-16T13:41:08.199 DEBUG:teuthology.orchestra.run.smithi124:> echo no | sudo yum reinstall kernel || true 2024-01-16T13:41:08.532 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2024-01-16T13:41:08.532 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2024-01-16T13:41:08.584 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:41:08.584 INFO:teuthology.orchestra.run.smithi124.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-16T13:41:08.584 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:41:08.893 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-16T13:41:08.893 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-16T13:41:08.902 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:01 ago on Tue 16 Jan 2024 01:41:07 PM UTC. 2024-01-16T13:41:08.979 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-01-16T13:41:08.980 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-01-16T13:41:08.980 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-01-16T13:41:08.981 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-16T13:41:08.982 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-16T13:41:09.034 INFO:teuthology.orchestra.run.smithi124.stderr:Error: No packages marked for reinstall. 2024-01-16T13:41:09.080 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum reinstall -y kernel || true 2024-01-16T13:41:09.132 DEBUG:teuthology.orchestra.run.smithi113:> echo no | sudo yum reinstall kernel || true 2024-01-16T13:41:09.413 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2024-01-16T13:41:09.413 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2024-01-16T13:41:09.467 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:41:09.467 INFO:teuthology.orchestra.run.smithi124.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-16T13:41:09.467 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:41:09.467 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-16T13:41:09.468 INFO:teuthology.orchestra.run.smithi113.stdout:Unable to read consumer identity 2024-01-16T13:41:09.525 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:41:09.525 INFO:teuthology.orchestra.run.smithi113.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-16T13:41:09.526 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:41:09.770 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:02 ago on Tue 16 Jan 2024 01:41:07 PM UTC. 2024-01-16T13:41:09.832 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-16T13:41:09.832 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-16T13:41:09.834 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:01 ago on Tue 16 Jan 2024 01:41:08 PM UTC. 2024-01-16T13:41:09.890 INFO:teuthology.orchestra.run.smithi124.stderr:Error: No packages marked for reinstall. 2024-01-16T13:41:09.924 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-16T13:41:09.935 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-16T13:41:09.938 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T13:41:09.973 INFO:teuthology.orchestra.run.smithi113.stderr:Error: No packages marked for reinstall. 2024-01-16T13:41:10.020 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum reinstall -y kernel || true 2024-01-16T13:41:10.088 INFO:teuthology.orchestra.run.smithi124.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:41:10.089 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:41:10.089 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-16T13:41:10.089 DEBUG:teuthology.parallel:result is None 2024-01-16T13:41:10.352 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-16T13:41:10.352 INFO:teuthology.orchestra.run.smithi113.stdout:Unable to read consumer identity 2024-01-16T13:41:10.408 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:41:10.409 INFO:teuthology.orchestra.run.smithi113.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-16T13:41:10.409 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:41:10.711 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:02 ago on Tue 16 Jan 2024 01:41:08 PM UTC. 2024-01-16T13:41:10.773 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-16T13:41:10.773 INFO:teuthology.orchestra.run.smithi113.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-16T13:41:10.812 INFO:teuthology.orchestra.run.smithi113.stderr:Error: No packages marked for reinstall. 2024-01-16T13:41:10.858 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T13:41:11.023 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:41:11.024 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-16T13:41:11.024 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-16T13:41:11.024 DEBUG:teuthology.parallel:result is None 2024-01-16T13:41:11.024 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T13:41:11.030 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T13:41:11.031 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T13:41:11.033 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T13:41:11.053 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T13:41:11.059 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T13:41:11.088 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T13:41:11.088 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T13:41:11.090 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T13:41:11.127 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T13:41:11.134 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T13:41:11.134 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T13:41:11.165 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T13:41:11.197 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T13:41:11.207 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T13:41:11.213 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T13:41:11.229 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T13:41:11.231 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T13:41:11.238 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T13:41:11.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T13:41:11.257 DEBUG:teuthology.orchestra.run.smithi124:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T13:41:11.303 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T13:41:11.312 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T13:41:11.312 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T13:41:11.324 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T13:41:11.363 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T13:41:11.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T13:41:11.511 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T13:41:11.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T13:41:11.595 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:41:11.595 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T13:41:11.661 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T13:41:11.701 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T13:41:11.763 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T13:41:11.847 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T13:41:11.897 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:41:11.897 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T13:41:11.967 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-01-16T13:41:11.969 DEBUG:teuthology.orchestra.run.smithi124:> sudo service rsyslog restart 2024-01-16T13:41:12.012 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T13:41:12.073 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T13:41:12.515 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T13:41:12.523 INFO:teuthology.task.internal:Starting timer... 2024-01-16T13:41:12.523 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T13:41:12.566 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T13:41:12.590 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-16T13:41:12.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-01-16T13:41:12.651 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-01-16T13:41:12.653 DEBUG:teuthology.orchestra.run.smithi124:> sudo service auditd rotate 2024-01-16T13:41:12.725 INFO:teuthology.orchestra.run.smithi124.stdout:Rotating logs: 2024-01-16T13:41:12.727 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T13:41:12.727 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-01-16T13:41:12.755 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-01-16T13:41:12.755 DEBUG:teuthology.orchestra.run.smithi124:> /usr/sbin/getenforce 2024-01-16T13:41:12.782 INFO:teuthology.orchestra.run.smithi124.stdout:Permissive 2024-01-16T13:41:12.782 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi113.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi124.front.sepia.ceph.com': 'permissive'} 2024-01-16T13:41:12.782 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T13:41:12.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:41:12.822 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T13:41:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:41:12.849 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T13:41:12.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-01-16T13:41:12.889 DEBUG:teuthology.orchestra.run.smithi124:> sudo /usr/sbin/setenforce permissive 2024-01-16T13:41:12.916 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T13:41:12.925 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-16T13:41:12.931 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-16T13:41:13.086 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T13:41:13.099 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T13:41:13.100 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi113.front.sepia.ceph.com,smithi124.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T13:48:19.034 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi113.front.sepia.ceph.com'), Remote(name='ubuntu@smithi124.front.sepia.ceph.com')] 2024-01-16T13:48:19.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T13:48:19.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T13:48:19.112 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-01-16T13:48:19.181 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T13:48:19.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T13:48:19.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T13:48:19.261 DEBUG:teuthology.orchestra.run.smithi124:> true 2024-01-16T13:48:19.329 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T13:48:19.330 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T13:48:19.339 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T13:48:19.339 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T13:48:19.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T13:48:19.341 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T13:48:19.342 DEBUG:teuthology.orchestra.run.smithi124:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T13:48:19.373 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T13:48:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T13:48:19.401 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T13:48:19.415 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T13:48:19.459 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-01-16T13:48:19.470 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-01-16T13:48:19.485 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T13:48:19.490 INFO:teuthology.orchestra.run.smithi124.stderr:sudo: ntpd: command not found 2024-01-16T13:48:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T13:48:19.501 INFO:teuthology.orchestra.run.smithi124.stdout:506 Cannot talk to daemon 2024-01-16T13:48:19.516 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T13:48:19.530 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T13:48:19.572 INFO:teuthology.orchestra.run.smithi113.stderr:bash: ntpq: command not found 2024-01-16T13:48:19.576 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T13:48:19.576 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-01-16T13:48:19.576 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.576 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.576 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.576 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:bash: ntpq: command not found 2024-01-16T13:48:19.602 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T13:48:19.602 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2024-01-16T13:48:19.602 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.602 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.602 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.602 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T13:48:19.603 INFO:teuthology.run_tasks:Running task install... 2024-01-16T13:48:19.611 DEBUG:teuthology.task.install:project ceph 2024-01-16T13:48:19.611 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}} 2024-01-16T13:48:19.611 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'} 2024-01-16T13:48:19.611 INFO:teuthology.task.install:Using flavor: default 2024-01-16T13:48:19.616 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T13:48:19.616 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-16T13:48:19.616 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T13:48:19.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T13:48:19.618 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T13:48:19.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T13:48:19.791 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/ 2024-01-16T13:48:19.792 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T13:48:19.931 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/ 2024-01-16T13:48:19.932 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T13:48:20.029 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T13:48:20.029 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:48:20.030 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T13:48:20.057 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T13:48:20.057 DEBUG:teuthology.orchestra.run.smithi124:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T13:48:20.124 DEBUG:teuthology.orchestra.run.smithi124:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T13:48:20.160 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T13:48:20.160 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:48:20.160 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T13:48:20.191 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T13:48:20.191 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T13:48:20.202 DEBUG:teuthology.orchestra.run.smithi124:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T13:48:20.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T13:48:20.268 INFO:teuthology.orchestra.run.smithi124.stdout:check_obsoletes = 1 2024-01-16T13:48:20.269 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum clean all 2024-01-16T13:48:20.342 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T13:48:20.409 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-01-16T13:48:20.411 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-01-16T13:48:20.643 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2024-01-16T13:48:20.787 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-16T13:48:20.988 INFO:teuthology.orchestra.run.smithi124.stdout:57 files removed 2024-01-16T13:48:21.037 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T13:48:21.087 INFO:teuthology.orchestra.run.smithi113.stdout:57 files removed 2024-01-16T13:48:21.134 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T13:48:21.373 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2024-01-16T13:48:21.469 INFO:teuthology.orchestra.run.smithi113.stdout:Updating Subscription Management repositories. 2024-01-16T13:48:22.253 INFO:teuthology.orchestra.run.smithi124.stdout:ceph packages for x86_64 187 kB/s | 81 kB 00:00 2024-01-16T13:48:22.372 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 180 kB/s | 81 kB 00:00 2024-01-16T13:48:22.681 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 35 kB/s | 9.5 kB 00:00 2024-01-16T13:48:22.854 INFO:teuthology.orchestra.run.smithi124.stdout:ceph noarch packages 17 kB/s | 9.5 kB 00:00 2024-01-16T13:48:23.089 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 3.9 kB/s | 1.5 kB 00:00 2024-01-16T13:48:23.455 INFO:teuthology.orchestra.run.smithi124.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2024-01-16T13:48:23.499 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 9.0 kB/s | 3.5 kB 00:00 2024-01-16T13:48:23.763 INFO:teuthology.orchestra.run.smithi124.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-01-16T13:48:26.143 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 16 MB 00:02 2024-01-16T13:48:26.875 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 16 MB 00:03 2024-01-16T13:48:29.708 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 316 kB/s | 24 kB 00:00 2024-01-16T13:48:30.472 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 510 kB/s | 24 kB 00:00 2024-01-16T13:48:30.793 INFO:teuthology.orchestra.run.smithi113.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 58 MB 00:01 2024-01-16T13:48:31.457 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 64 MB/s | 58 MB 00:00 2024-01-16T13:48:40.860 INFO:teuthology.orchestra.run.smithi113.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 27 MB/s | 8.8 MB 00:00 2024-01-16T13:48:42.224 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 MB/s | 65 MB 00:00 2024-01-16T13:48:43.577 INFO:teuthology.orchestra.run.smithi113.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 65 MB 00:01 2024-01-16T13:48:52.242 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 26 MB/s | 8.8 MB 00:00 2024-01-16T13:48:58.199 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-16T13:48:58.199 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-16T13:48:58.217 INFO:teuthology.orchestra.run.smithi113.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-16T13:48:58.219 INFO:teuthology.orchestra.run.smithi113.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-16T13:48:58.220 INFO:teuthology.orchestra.run.smithi113.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-16T13:48:58.222 INFO:teuthology.orchestra.run.smithi113.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-16T13:48:58.225 INFO:teuthology.orchestra.run.smithi113.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-16T13:48:58.229 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-16T13:48:58.230 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-16T13:48:58.434 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout:=================================================================================================================================== 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout:=================================================================================================================================== 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 6.4 k 2024-01-16T13:48:58.448 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 5.4 M 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 919 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 171 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 1.6 M 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 133 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 1.8 M 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 7.4 M 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 50 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 14 M 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 49 M 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 262 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 410 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 263 k 2024-01-16T13:48:58.449 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 18 k 2024-01-16T13:48:58.450 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-16T13:48:58.450 INFO:teuthology.orchestra.run.smithi113.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-16T13:48:58.450 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-16T13:48:58.450 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 31 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 839 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 139 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 197 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 371 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 365 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 107 k 2024-01-16T13:48:58.451 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 93 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 3.6 M 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 181 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 3.8 M 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 3.7 M 2024-01-16T13:48:58.452 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 23 M 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 28 k 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 2.3 M 2024-01-16T13:48:58.453 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 246 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 4.6 M 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 18 M 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 15 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 24 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 183 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-16T13:48:58.454 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 517 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 6.9 M 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-16T13:48:58.455 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel x86_64 5.3.4-12.el8 codeready-builder-for-rhel-8-x86_64-rpms 30 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-16T13:48:58.456 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 45 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 129 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T13:48:58.457 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-16T13:48:58.458 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T13:48:58.459 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T13:48:58.460 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout:Enabling module streams: 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql 10 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout:=================================================================================================================================== 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout:Install 131 Packages 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 16 Packages 2024-01-16T13:48:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:48:58.462 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 218 M 2024-01-16T13:48:58.462 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-01-16T13:48:59.226 INFO:teuthology.orchestra.run.smithi124.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-16T13:48:59.227 INFO:teuthology.orchestra.run.smithi124.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-16T13:48:59.245 INFO:teuthology.orchestra.run.smithi124.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-16T13:48:59.246 INFO:teuthology.orchestra.run.smithi124.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-16T13:48:59.248 INFO:teuthology.orchestra.run.smithi124.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-16T13:48:59.249 INFO:teuthology.orchestra.run.smithi124.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-16T13:48:59.252 INFO:teuthology.orchestra.run.smithi124.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-16T13:48:59.256 INFO:teuthology.orchestra.run.smithi124.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-16T13:48:59.257 INFO:teuthology.orchestra.run.smithi124.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-16T13:48:59.467 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2024-01-16T13:48:59.481 INFO:teuthology.orchestra.run.smithi124.stdout:=================================================================================================================================== 2024-01-16T13:48:59.481 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2024-01-16T13:48:59.481 INFO:teuthology.orchestra.run.smithi124.stdout:=================================================================================================================================== 2024-01-16T13:48:59.481 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2024-01-16T13:48:59.481 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-16T13:48:59.481 INFO:teuthology.orchestra.run.smithi124.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 6.4 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 5.4 M 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 919 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 171 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 1.6 M 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 133 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 1.8 M 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 7.4 M 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 50 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 14 M 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 49 M 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 262 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 410 k 2024-01-16T13:48:59.482 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 263 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 18 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 31 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 839 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 139 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 197 k 2024-01-16T13:48:59.483 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 371 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 365 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 107 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 93 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 3.6 M 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 181 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrading: 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 3.8 M 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 3.7 M 2024-01-16T13:48:59.484 INFO:teuthology.orchestra.run.smithi124.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 23 M 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 28 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 2.3 M 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 246 k 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 4.6 M 2024-01-16T13:48:59.485 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 18 M 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el8 ceph-noarch 15 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 24 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 183 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 517 k 2024-01-16T13:48:59.486 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 6.9 M 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: lua-devel x86_64 5.3.4-12.el8 codeready-builder-for-rhel-8-x86_64-rpms 30 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-16T13:48:59.487 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 45 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el8 ceph 129 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T13:48:59.488 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-16T13:48:59.489 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T13:48:59.490 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout:Enabling module streams: 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql 10 2024-01-16T13:48:59.491 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout:=================================================================================================================================== 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout:Install 131 Packages 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrade 16 Packages 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 218 M 2024-01-16T13:48:59.492 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2024-01-16T13:49:05.742 INFO:teuthology.orchestra.run.smithi113.stdout:(1/147): ceph-18.0.0-7629.g73a4532f.el8.x86_64. 909 B/s | 6.4 kB 00:07 2024-01-16T13:49:05.992 INFO:teuthology.orchestra.run.smithi124.stdout:(1/147): ceph-18.0.0-7629.g73a4532f.el8.x86_64. 1.0 kB/s | 6.4 kB 00:06 2024-01-16T13:49:06.159 INFO:teuthology.orchestra.run.smithi113.stdout:(2/147): ceph-base-18.0.0-7629.g73a4532f.el8.x8 718 kB/s | 5.4 MB 00:07 2024-01-16T13:49:06.218 INFO:teuthology.orchestra.run.smithi113.stdout:(3/147): ceph-fuse-18.0.0-7629.g73a4532f.el8.x8 1.9 MB/s | 919 kB 00:00 2024-01-16T13:49:06.243 INFO:teuthology.orchestra.run.smithi113.stdout:(4/147): ceph-immutable-object-cache-18.0.0-762 2.0 MB/s | 171 kB 00:00 2024-01-16T13:49:06.543 INFO:teuthology.orchestra.run.smithi113.stdout:(5/147): ceph-mgr-18.0.0-7629.g73a4532f.el8.x86 5.4 MB/s | 1.6 MB 00:00 2024-01-16T13:49:06.943 INFO:teuthology.orchestra.run.smithi113.stdout:(6/147): ceph-mds-18.0.0-7629.g73a4532f.el8.x86 3.2 MB/s | 2.3 MB 00:00 2024-01-16T13:49:07.192 INFO:teuthology.orchestra.run.smithi124.stdout:(2/147): ceph-fuse-18.0.0-7629.g73a4532f.el8.x8 766 kB/s | 919 kB 00:01 2024-01-16T13:49:07.467 INFO:teuthology.orchestra.run.smithi124.stdout:(3/147): ceph-immutable-object-cache-18.0.0-762 623 kB/s | 171 kB 00:00 2024-01-16T13:49:07.550 INFO:teuthology.orchestra.run.smithi113.stdout:(7/147): ceph-mon-18.0.0-7629.g73a4532f.el8.x86 4.6 MB/s | 4.6 MB 00:01 2024-01-16T13:49:08.434 INFO:teuthology.orchestra.run.smithi124.stdout:(4/147): ceph-common-18.0.0-7629.g73a4532f.el8. 2.6 MB/s | 23 MB 00:08 2024-01-16T13:49:08.618 INFO:teuthology.orchestra.run.smithi124.stdout:(5/147): ceph-mgr-18.0.0-7629.g73a4532f.el8.x86 8.9 MB/s | 1.6 MB 00:00 2024-01-16T13:49:08.684 INFO:teuthology.orchestra.run.smithi124.stdout:(6/147): ceph-base-18.0.0-7629.g73a4532f.el8.x8 616 kB/s | 5.4 MB 00:08 2024-01-16T13:49:08.727 INFO:teuthology.orchestra.run.smithi113.stdout:(8/147): ceph-common-18.0.0-7629.g73a4532f.el8. 2.2 MB/s | 23 MB 00:10 2024-01-16T13:49:08.852 INFO:teuthology.orchestra.run.smithi113.stdout:(9/147): ceph-selinux-18.0.0-7629.g73a4532f.el8 195 kB/s | 24 kB 00:00 2024-01-16T13:49:08.993 INFO:teuthology.orchestra.run.smithi124.stdout:(7/147): ceph-mon-18.0.0-7629.g73a4532f.el8.x86 12 MB/s | 4.6 MB 00:00 2024-01-16T13:49:09.254 INFO:teuthology.orchestra.run.smithi113.stdout:(10/147): ceph-radosgw-18.0.0-7629.g73a4532f.el 8.1 MB/s | 14 MB 00:01 2024-01-16T13:49:09.353 INFO:teuthology.orchestra.run.smithi113.stdout:(11/147): cephfs-mirror-18.0.0-7629.g73a4532f.e 2.6 MB/s | 263 kB 00:00 2024-01-16T13:49:09.420 INFO:teuthology.orchestra.run.smithi113.stdout:(12/147): libcephfs-devel-18.0.0-7629.g73a4532f 466 kB/s | 31 kB 00:00 2024-01-16T13:49:09.628 INFO:teuthology.orchestra.run.smithi113.stdout:(13/147): libcephfs2-18.0.0-7629.g73a4532f.el8. 3.9 MB/s | 839 kB 00:00 2024-01-16T13:49:09.729 INFO:teuthology.orchestra.run.smithi113.stdout:(14/147): libcephsqlite-18.0.0-7629.g73a4532f.e 1.8 MB/s | 183 kB 00:00 2024-01-16T13:49:09.837 INFO:teuthology.orchestra.run.smithi113.stdout:(15/147): librados-devel-18.0.0-7629.g73a4532f. 1.3 MB/s | 139 kB 00:00 2024-01-16T13:49:10.002 INFO:teuthology.orchestra.run.smithi124.stdout:(8/147): ceph-radosgw-18.0.0-7629.g73a4532f.el8 14 MB/s | 14 MB 00:01 2024-01-16T13:49:10.043 INFO:teuthology.orchestra.run.smithi124.stdout:(9/147): ceph-selinux-18.0.0-7629.g73a4532f.el8 588 kB/s | 24 kB 00:00 2024-01-16T13:49:10.172 INFO:teuthology.orchestra.run.smithi113.stdout:(16/147): libradosstriper1-18.0.0-7629.g73a4532 1.5 MB/s | 517 kB 00:00 2024-01-16T13:49:10.394 INFO:teuthology.orchestra.run.smithi124.stdout:(10/147): ceph-mds-18.0.0-7629.g73a4532f.el8.x8 809 kB/s | 2.3 MB 00:02 2024-01-16T13:49:11.221 INFO:teuthology.orchestra.run.smithi113.stdout:(17/147): librgw2-18.0.0-7629.g73a4532f.el8.x86 6.6 MB/s | 6.9 MB 00:01 2024-01-16T13:49:11.236 INFO:teuthology.orchestra.run.smithi124.stdout:(11/147): cephfs-mirror-18.0.0-7629.g73a4532f.e 312 kB/s | 263 kB 00:00 2024-01-16T13:49:11.296 INFO:teuthology.orchestra.run.smithi113.stdout:(18/147): python3-ceph-argparse-18.0.0-7629.g73 603 kB/s | 45 kB 00:00 2024-01-16T13:49:11.369 INFO:teuthology.orchestra.run.smithi124.stdout:(12/147): libcephfs-devel-18.0.0-7629.g73a4532f 233 kB/s | 31 kB 00:00 2024-01-16T13:49:11.554 INFO:teuthology.orchestra.run.smithi113.stdout:(19/147): python3-ceph-common-18.0.0-7629.g73a4 498 kB/s | 129 kB 00:00 2024-01-16T13:49:11.655 INFO:teuthology.orchestra.run.smithi113.stdout:(20/147): python3-cephfs-18.0.0-7629.g73a4532f. 1.9 MB/s | 197 kB 00:00 2024-01-16T13:49:11.790 INFO:teuthology.orchestra.run.smithi113.stdout:(21/147): python3-rados-18.0.0-7629.g73a4532f.e 2.7 MB/s | 371 kB 00:00 2024-01-16T13:49:11.888 INFO:teuthology.orchestra.run.smithi113.stdout:(22/147): python3-rbd-18.0.0-7629.g73a4532f.el8 3.6 MB/s | 365 kB 00:00 2024-01-16T13:49:11.964 INFO:teuthology.orchestra.run.smithi113.stdout:(23/147): python3-rgw-18.0.0-7629.g73a4532f.el8 1.4 MB/s | 107 kB 00:00 2024-01-16T13:49:12.081 INFO:teuthology.orchestra.run.smithi113.stdout:(24/147): rbd-fuse-18.0.0-7629.g73a4532f.el8.x8 802 kB/s | 93 kB 00:00 2024-01-16T13:49:12.461 INFO:teuthology.orchestra.run.smithi124.stdout:(13/147): libcephfs2-18.0.0-7629.g73a4532f.el8. 769 kB/s | 839 kB 00:01 2024-01-16T13:49:12.648 INFO:teuthology.orchestra.run.smithi113.stdout:(25/147): rbd-mirror-18.0.0-7629.g73a4532f.el8. 6.3 MB/s | 3.6 MB 00:00 2024-01-16T13:49:12.753 INFO:teuthology.orchestra.run.smithi124.stdout:(14/147): libcephsqlite-18.0.0-7629.g73a4532f.e 628 kB/s | 183 kB 00:00 2024-01-16T13:49:12.757 INFO:teuthology.orchestra.run.smithi113.stdout:(26/147): rbd-nbd-18.0.0-7629.g73a4532f.el8.x86 1.6 MB/s | 181 kB 00:00 2024-01-16T13:49:12.832 INFO:teuthology.orchestra.run.smithi113.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7629.g 372 kB/s | 28 kB 00:00 2024-01-16T13:49:12.916 INFO:teuthology.orchestra.run.smithi113.stdout:(28/147): ceph-mgr-cephadm-18.0.0-7629.g73a4532 1.6 MB/s | 133 kB 00:00 2024-01-16T13:49:12.928 INFO:teuthology.orchestra.run.smithi124.stdout:(15/147): librados-devel-18.0.0-7629.g73a4532f. 796 kB/s | 139 kB 00:00 2024-01-16T13:49:13.232 INFO:teuthology.orchestra.run.smithi113.stdout:(29/147): ceph-mgr-dashboard-18.0.0-7629.g73a45 5.6 MB/s | 1.8 MB 00:00 2024-01-16T13:49:13.912 INFO:teuthology.orchestra.run.smithi124.stdout:(16/147): ceph-test-18.0.0-7629.g73a4532f.el8.x 13 MB/s | 49 MB 00:03 2024-01-16T13:49:13.937 INFO:teuthology.orchestra.run.smithi124.stdout:(17/147): libradosstriper1-18.0.0-7629.g73a4532 513 kB/s | 517 kB 00:01 2024-01-16T13:49:13.999 INFO:teuthology.orchestra.run.smithi113.stdout:(30/147): ceph-osd-18.0.0-7629.g73a4532f.el8.x8 2.6 MB/s | 18 MB 00:07 2024-01-16T13:49:14.100 INFO:teuthology.orchestra.run.smithi113.stdout:(31/147): ceph-mgr-diskprediction-local-18.0.0- 8.5 MB/s | 7.4 MB 00:00 2024-01-16T13:49:14.104 INFO:teuthology.orchestra.run.smithi124.stdout:(18/147): python3-ceph-argparse-18.0.0-7629.g73 269 kB/s | 45 kB 00:00 2024-01-16T13:49:14.125 INFO:teuthology.orchestra.run.smithi113.stdout:(32/147): ceph-mgr-modules-core-18.0.0-7629.g73 1.9 MB/s | 246 kB 00:00 2024-01-16T13:49:14.158 INFO:teuthology.orchestra.run.smithi113.stdout:(33/147): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 858 kB/s | 50 kB 00:00 2024-01-16T13:49:14.192 INFO:teuthology.orchestra.run.smithi113.stdout:(34/147): ceph-prometheus-alerts-18.0.0-7629.g7 224 kB/s | 15 kB 00:00 2024-01-16T13:49:14.287 INFO:teuthology.orchestra.run.smithi124.stdout:(19/147): python3-ceph-common-18.0.0-7629.g73a4 702 kB/s | 129 kB 00:00 2024-01-16T13:49:14.317 INFO:teuthology.orchestra.run.smithi113.stdout:(35/147): ceph-volume-18.0.0-7629.g73a4532f.el8 1.6 MB/s | 262 kB 00:00 2024-01-16T13:49:14.362 INFO:teuthology.orchestra.run.smithi124.stdout:(20/147): librgw2-18.0.0-7629.g73a4532f.el8.x86 15 MB/s | 6.9 MB 00:00 2024-01-16T13:49:14.368 INFO:teuthology.orchestra.run.smithi113.stdout:(36/147): cephfs-top-18.0.0-7629.g73a4532f.el8. 363 kB/s | 18 kB 00:00 2024-01-16T13:49:14.401 INFO:teuthology.orchestra.run.smithi113.stdout:(37/147): cephadm-18.0.0-7629.g73a4532f.el8.noa 1.9 MB/s | 410 kB 00:00 2024-01-16T13:49:14.412 INFO:teuthology.orchestra.run.smithi124.stdout:(21/147): python3-rados-18.0.0-7629.g73a4532f.e 7.3 MB/s | 371 kB 00:00 2024-01-16T13:49:14.496 INFO:teuthology.orchestra.run.smithi124.stdout:(22/147): python3-rbd-18.0.0-7629.g73a4532f.el8 4.3 MB/s | 365 kB 00:00 2024-01-16T13:49:14.521 INFO:teuthology.orchestra.run.smithi124.stdout:(23/147): python3-cephfs-18.0.0-7629.g73a4532f. 841 kB/s | 197 kB 00:00 2024-01-16T13:49:14.547 INFO:teuthology.orchestra.run.smithi124.stdout:(24/147): python3-rgw-18.0.0-7629.g73a4532f.el8 2.1 MB/s | 107 kB 00:00 2024-01-16T13:49:14.689 INFO:teuthology.orchestra.run.smithi124.stdout:(25/147): rbd-fuse-18.0.0-7629.g73a4532f.el8.x8 560 kB/s | 93 kB 00:00 2024-01-16T13:49:14.781 INFO:teuthology.orchestra.run.smithi124.stdout:(26/147): rbd-mirror-18.0.0-7629.g73a4532f.el8. 15 MB/s | 3.6 MB 00:00 2024-01-16T13:49:14.810 INFO:teuthology.orchestra.run.smithi113.stdout:(38/147): liboath-2.6.2-3.el8.x86_64.rpm 144 kB/s | 59 kB 00:00 2024-01-16T13:49:14.823 INFO:teuthology.orchestra.run.smithi124.stdout:(27/147): ceph-grafana-dashboards-18.0.0-7629.g 674 kB/s | 28 kB 00:00 2024-01-16T13:49:14.873 INFO:teuthology.orchestra.run.smithi124.stdout:(28/147): ceph-mgr-cephadm-18.0.0-7629.g73a4532 2.6 MB/s | 133 kB 00:00 2024-01-16T13:49:14.923 INFO:teuthology.orchestra.run.smithi124.stdout:(29/147): rbd-nbd-18.0.0-7629.g73a4532f.el8.x86 773 kB/s | 181 kB 00:00 2024-01-16T13:49:14.952 INFO:teuthology.orchestra.run.smithi113.stdout:(39/147): gperftools-libs-2.7-9.el8.x86_64.rpm 525 kB/s | 306 kB 00:00 2024-01-16T13:49:14.977 INFO:teuthology.orchestra.run.smithi113.stdout:(40/147): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2024-01-16T13:49:14.990 INFO:teuthology.orchestra.run.smithi124.stdout:(30/147): ceph-mgr-dashboard-18.0.0-7629.g73a45 15 MB/s | 1.8 MB 00:00 2024-01-16T13:49:15.066 INFO:teuthology.orchestra.run.smithi124.stdout:(31/147): ceph-mgr-modules-core-18.0.0-7629.g73 3.2 MB/s | 246 kB 00:00 2024-01-16T13:49:15.094 INFO:teuthology.orchestra.run.smithi113.stdout:(41/147): luarocks-3.5.0-1.el8.noarch.rpm 1.1 MB/s | 167 kB 00:00 2024-01-16T13:49:15.099 INFO:teuthology.orchestra.run.smithi124.stdout:(32/147): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 1.5 MB/s | 50 kB 00:00 2024-01-16T13:49:15.133 INFO:teuthology.orchestra.run.smithi124.stdout:(33/147): ceph-prometheus-alerts-18.0.0-7629.g7 448 kB/s | 15 kB 00:00 2024-01-16T13:49:15.174 INFO:teuthology.orchestra.run.smithi124.stdout:(34/147): ceph-volume-18.0.0-7629.g73a4532f.el8 6.2 MB/s | 262 kB 00:00 2024-01-16T13:49:15.211 INFO:teuthology.orchestra.run.smithi113.stdout:(42/147): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 374 kB/s | 44 kB 00:00 2024-01-16T13:49:15.225 INFO:teuthology.orchestra.run.smithi124.stdout:(35/147): cephadm-18.0.0-7629.g73a4532f.el8.noa 8.0 MB/s | 410 kB 00:00 2024-01-16T13:49:15.267 INFO:teuthology.orchestra.run.smithi124.stdout:(36/147): cephfs-top-18.0.0-7629.g73a4532f.el8. 436 kB/s | 18 kB 00:00 2024-01-16T13:49:15.353 INFO:teuthology.orchestra.run.smithi113.stdout:(43/147): python3-asyncssh-2.7.0-2.el8.noarch.r 1.2 MB/s | 476 kB 00:00 2024-01-16T13:49:15.370 INFO:teuthology.orchestra.run.smithi113.stdout:(44/147): python3-beautifulsoup4-4.6.3-2.el8.1. 1.1 MB/s | 185 kB 00:00 2024-01-16T13:49:15.470 INFO:teuthology.orchestra.run.smithi113.stdout:(45/147): python3-cachetools-3.1.1-4.el8.noarch 282 kB/s | 33 kB 00:00 2024-01-16T13:49:15.495 INFO:teuthology.orchestra.run.smithi113.stdout:(46/147): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2024-01-16T13:49:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:(47/147): python3-cheroot-8.5.2-1.el8.noarch.rp 795 kB/s | 173 kB 00:00 2024-01-16T13:49:15.730 INFO:teuthology.orchestra.run.smithi113.stdout:(48/147): python3-cherrypy-18.4.0-1.el8.noarch. 1.6 MB/s | 384 kB 00:00 2024-01-16T13:49:15.813 INFO:teuthology.orchestra.run.smithi113.stdout:(49/147): python3-google-auth-1.1.1-10.el8.noar 650 kB/s | 81 kB 00:00 2024-01-16T13:49:15.838 INFO:teuthology.orchestra.run.smithi113.stdout:(50/147): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-01-16T13:49:15.948 INFO:teuthology.orchestra.run.smithi113.stdout:(51/147): python3-jaraco-functools-2.0-4.el8.no 131 kB/s | 18 kB 00:00 2024-01-16T13:49:16.048 INFO:teuthology.orchestra.run.smithi113.stdout:(52/147): python3-logutils-0.3.5-11.el8.noarch. 488 kB/s | 49 kB 00:00 2024-01-16T13:49:16.098 INFO:teuthology.orchestra.run.smithi113.stdout:(53/147): python3-kubernetes-11.0.0-6.el8.noarc 4.5 MB/s | 1.2 MB 00:00 2024-01-16T13:49:16.199 INFO:teuthology.orchestra.run.smithi113.stdout:(54/147): python3-more-itertools-7.2.0-3.el8.no 390 kB/s | 59 kB 00:00 2024-01-16T13:49:16.224 INFO:teuthology.orchestra.run.smithi113.stdout:(55/147): python3-natsort-7.1.1-2.el8.noarch.rp 481 kB/s | 60 kB 00:00 2024-01-16T13:49:16.324 INFO:teuthology.orchestra.run.smithi113.stdout:(56/147): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-01-16T13:49:16.349 INFO:teuthology.orchestra.run.smithi113.stdout:(57/147): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-01-16T13:49:16.454 INFO:teuthology.orchestra.run.smithi113.stdout:(58/147): python3-repoze-lru-0.7-6.el8.noarch.r 260 kB/s | 34 kB 00:00 2024-01-16T13:49:16.483 INFO:teuthology.orchestra.run.smithi113.stdout:(59/147): python3-routes-2.4.1-12.el8.noarch.rp 1.4 MB/s | 196 kB 00:00 2024-01-16T13:49:16.559 INFO:teuthology.orchestra.run.smithi113.stdout:(60/147): python3-rsa-4.9-2.el8.noarch.rpm 615 kB/s | 64 kB 00:00 2024-01-16T13:49:16.576 INFO:teuthology.orchestra.run.smithi113.stdout:(61/147): python3-simplegeneric-0.8.1-17.el8.no 203 kB/s | 19 kB 00:00 2024-01-16T13:49:16.667 INFO:teuthology.orchestra.run.smithi113.stdout:(62/147): python3-singledispatch-3.4.0.3-18.el8 220 kB/s | 24 kB 00:00 2024-01-16T13:49:16.684 INFO:teuthology.orchestra.run.smithi113.stdout:(63/147): python3-tempora-1.14.1-5.el8.noarch.r 269 kB/s | 29 kB 00:00 2024-01-16T13:49:16.760 INFO:teuthology.orchestra.run.smithi113.stdout:(64/147): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2024-01-16T13:49:16.827 INFO:teuthology.orchestra.run.smithi113.stdout:(65/147): python3-waitress-1.4.3-1.el8.noarch.r 1.8 MB/s | 255 kB 00:00 2024-01-16T13:49:16.877 INFO:teuthology.orchestra.run.smithi113.stdout:(66/147): python3-webob-1.8.5-1.el8.1.noarch.rp 2.1 MB/s | 251 kB 00:00 2024-01-16T13:49:16.919 INFO:teuthology.orchestra.run.smithi113.stdout:(67/147): python3-websocket-client-0.56.0-5.el8 665 kB/s | 61 kB 00:00 2024-01-16T13:49:16.967 INFO:teuthology.orchestra.run.smithi124.stdout:(37/147): gperftools-libs-2.7-9.el8.x86_64.rpm 180 kB/s | 306 kB 00:01 2024-01-16T13:49:17.003 INFO:teuthology.orchestra.run.smithi113.stdout:(68/147): python3-webtest-2.0.33-1.el8.noarch.r 680 kB/s | 85 kB 00:00 2024-01-16T13:49:17.028 INFO:teuthology.orchestra.run.smithi113.stdout:(69/147): python3-zc-lockfile-2.0-2.el8.noarch. 211 kB/s | 23 kB 00:00 2024-01-16T13:49:17.103 INFO:teuthology.orchestra.run.smithi113.stdout:(70/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 928 kB/s | 70 kB 00:00 2024-01-16T13:49:17.170 INFO:teuthology.orchestra.run.smithi113.stdout:(71/147): python3-jsonpointer-1.10-11.el8.noarc 301 kB/s | 20 kB 00:00 2024-01-16T13:49:17.221 INFO:teuthology.orchestra.run.smithi113.stdout:(72/147): python3-prettytable-0.7.2-14.el8.noar 878 kB/s | 44 kB 00:00 2024-01-16T13:49:17.292 INFO:teuthology.orchestra.run.smithi113.stdout:(73/147): thrift-0.13.0-2.el8.x86_64.rpm 6.0 MB/s | 1.7 MB 00:00 2024-01-16T13:49:17.321 INFO:teuthology.orchestra.run.smithi113.stdout:(74/147): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2024-01-16T13:49:17.380 INFO:teuthology.orchestra.run.smithi113.stdout:(75/147): python3-jsonpatch-1.21-2.el8.noarch.r 303 kB/s | 27 kB 00:00 2024-01-16T13:49:17.405 INFO:teuthology.orchestra.run.smithi113.stdout:(76/147): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2024-01-16T13:49:17.409 INFO:teuthology.orchestra.run.smithi124.stdout:(38/147): ceph-osd-18.0.0-7629.g73a4532f.el8.x8 2.1 MB/s | 18 MB 00:08 2024-01-16T13:49:17.434 INFO:teuthology.orchestra.run.smithi124.stdout:(39/147): liboath-2.6.2-3.el8.x86_64.rpm 126 kB/s | 59 kB 00:00 2024-01-16T13:49:17.439 INFO:teuthology.orchestra.run.smithi113.stdout:(77/147): python3-werkzeug-0.12.2-4.el8.noarch. 7.6 MB/s | 457 kB 00:00 2024-01-16T13:49:17.473 INFO:teuthology.orchestra.run.smithi113.stdout:(78/147): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2024-01-16T13:49:17.506 INFO:teuthology.orchestra.run.smithi113.stdout:(79/147): flex-2.6.1-9.el8.x86_64.rpm 4.7 MB/s | 320 kB 00:00 2024-01-16T13:49:17.562 INFO:teuthology.orchestra.run.smithi113.stdout:(80/147): libtool-2.4.6-25.el8.x86_64.rpm 7.8 MB/s | 709 kB 00:00 2024-01-16T13:49:17.590 INFO:teuthology.orchestra.run.smithi113.stdout:(81/147): python3-markupsafe-0.23-19.el8.x86_64 464 kB/s | 39 kB 00:00 2024-01-16T13:49:17.615 INFO:teuthology.orchestra.run.smithi113.stdout:(82/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.8 MB/s | 259 kB 00:00 2024-01-16T13:49:17.641 INFO:teuthology.orchestra.run.smithi113.stdout:(83/147): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-01-16T13:49:17.666 INFO:teuthology.orchestra.run.smithi113.stdout:(84/147): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-01-16T13:49:17.683 INFO:teuthology.orchestra.run.smithi113.stdout:(85/147): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2024-01-16T13:49:17.742 INFO:teuthology.orchestra.run.smithi113.stdout:(86/147): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2024-01-16T13:49:17.810 INFO:teuthology.orchestra.run.smithi124.stdout:(40/147): luarocks-3.5.0-1.el8.noarch.rpm 445 kB/s | 167 kB 00:00 2024-01-16T13:49:18.209 INFO:teuthology.orchestra.run.smithi113.stdout:(87/147): ceph-test-18.0.0-7629.g73a4532f.el8.x 5.3 MB/s | 49 MB 00:09 2024-01-16T13:49:18.317 INFO:teuthology.orchestra.run.smithi113.stdout:(88/147): python3-babel-2.5.1-7.el8.noarch.rpm 7.4 MB/s | 4.8 MB 00:00 2024-01-16T13:49:18.326 INFO:teuthology.orchestra.run.smithi124.stdout:(41/147): python3-asyncssh-2.7.0-2.el8.noarch.r 921 kB/s | 476 kB 00:00 2024-01-16T13:49:18.342 INFO:teuthology.orchestra.run.smithi113.stdout:(89/147): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2024-01-16T13:49:18.359 INFO:teuthology.orchestra.run.smithi113.stdout:(90/147): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2024-01-16T13:49:18.456 INFO:teuthology.orchestra.run.smithi113.stdout:(91/147): python3-numpy-1.14.3-10.el8.x86_64.rp 5.2 MB/s | 3.7 MB 00:00 2024-01-16T13:49:18.493 INFO:teuthology.orchestra.run.smithi113.stdout:(92/147): librdkafka-0.11.4-3.el8.x86_64.rpm 2.3 MB/s | 354 kB 00:00 2024-01-16T13:49:18.519 INFO:teuthology.orchestra.run.smithi113.stdout:(93/147): python3-mako-1.0.6-14.el8.noarch.rpm 2.4 MB/s | 157 kB 00:00 2024-01-16T13:49:18.585 INFO:teuthology.orchestra.run.smithi124.stdout:(42/147): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 169 kB/s | 44 kB 00:00 2024-01-16T13:49:18.652 INFO:teuthology.orchestra.run.smithi113.stdout:(94/147): openblas-threads-0.3.15-6.el8.x86_64. 18 MB/s | 5.2 MB 00:00 2024-01-16T13:49:18.678 INFO:teuthology.orchestra.run.smithi113.stdout:(95/147): automake-1.16.1-8.el8.noarch.rpm 4.4 MB/s | 714 kB 00:00 2024-01-16T13:49:18.685 INFO:teuthology.orchestra.run.smithi124.stdout:(43/147): libunwind-1.3.1-3.el8.x86_64.rpm 59 kB/s | 75 kB 00:01 2024-01-16T13:49:18.742 INFO:teuthology.orchestra.run.smithi113.stdout:(96/147): libpq-13.11-1.el8.x86_64.rpm 2.2 MB/s | 198 kB 00:00 2024-01-16T13:49:18.778 INFO:teuthology.orchestra.run.smithi113.stdout:(97/147): jq-1.6-7.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2024-01-16T13:49:18.795 INFO:teuthology.orchestra.run.smithi113.stdout:(98/147): python3-pytz-2017.2-11.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2024-01-16T13:49:18.861 INFO:teuthology.orchestra.run.smithi124.stdout:(44/147): python3-beautifulsoup4-4.6.3-2.el8.1. 672 kB/s | 185 kB 00:00 2024-01-16T13:49:18.870 INFO:teuthology.orchestra.run.smithi113.stdout:(99/147): openblas-0.3.15-6.el8.x86_64.rpm 13 MB/s | 5.0 MB 00:00 2024-01-16T13:49:18.912 INFO:teuthology.orchestra.run.smithi113.stdout:(100/147): postgresql-contrib-10.23-3.module+el 6.8 MB/s | 811 kB 00:00 2024-01-16T13:49:18.954 INFO:teuthology.orchestra.run.smithi113.stdout:(101/147): postgresql-10.23-3.module+el8.9.0+20 8.6 MB/s | 1.5 MB 00:00 2024-01-16T13:49:19.002 INFO:teuthology.orchestra.run.smithi124.stdout:(45/147): python3-cachetools-3.1.1-4.el8.noarch 104 kB/s | 33 kB 00:00 2024-01-16T13:49:19.030 INFO:teuthology.orchestra.run.smithi113.stdout:(102/147): lua-devel-5.3.4-12.el8.x86_64.rpm 400 kB/s | 30 kB 00:00 2024-01-16T13:49:19.080 INFO:teuthology.orchestra.run.smithi113.stdout:(103/147): libverto-devel-0.3.0-5.el8.x86_64.rp 359 kB/s | 18 kB 00:00 2024-01-16T13:49:19.111 INFO:teuthology.orchestra.run.smithi124.stdout:(46/147): python3-certifi-2018.10.15-7.el8.noar 65 kB/s | 16 kB 00:00 2024-01-16T13:49:19.155 INFO:teuthology.orchestra.run.smithi113.stdout:(104/147): postgresql-server-10.23-3.module+el8 18 MB/s | 5.1 MB 00:00 2024-01-16T13:49:19.189 INFO:teuthology.orchestra.run.smithi113.stdout:(105/147): python3-jwt-1.6.1-2.el8.noarch.rpm 392 kB/s | 43 kB 00:00 2024-01-16T13:49:19.214 INFO:teuthology.orchestra.run.smithi113.stdout:(106/147): python3-oauthlib-2.1.0-1.el8.noarch. 2.6 MB/s | 155 kB 00:00 2024-01-16T13:49:19.231 INFO:teuthology.orchestra.run.smithi113.stdout:(107/147): python3-pycparser-2.14-14.el8.noarch 2.6 MB/s | 109 kB 00:00 2024-01-16T13:49:19.299 INFO:teuthology.orchestra.run.smithi113.stdout:(108/147): python3-requests-oauthlib-1.0.0-1.el 512 kB/s | 43 kB 00:00 2024-01-16T13:49:19.332 INFO:teuthology.orchestra.run.smithi113.stdout:(109/147): libaio-devel-0.3.112-1.el8.x86_64.rp 187 kB/s | 19 kB 00:00 2024-01-16T13:49:19.357 INFO:teuthology.orchestra.run.smithi113.stdout:(110/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-01-16T13:49:19.391 INFO:teuthology.orchestra.run.smithi113.stdout:(111/147): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2024-01-16T13:49:19.416 INFO:teuthology.orchestra.run.smithi113.stdout:(112/147): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2024-01-16T13:49:19.428 INFO:teuthology.orchestra.run.smithi124.stdout:(47/147): python3-cherrypy-18.4.0-1.el8.noarch. 1.2 MB/s | 384 kB 00:00 2024-01-16T13:49:19.441 INFO:teuthology.orchestra.run.smithi113.stdout:(113/147): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-01-16T13:49:19.467 INFO:teuthology.orchestra.run.smithi113.stdout:(114/147): libselinux-devel-2.9-5.el8.x86_64.rp 3.9 MB/s | 200 kB 00:00 2024-01-16T13:49:19.492 INFO:teuthology.orchestra.run.smithi113.stdout:(115/147): keyutils-libs-devel-1.5.10-9.el8.x86 957 kB/s | 48 kB 00:00 2024-01-16T13:49:19.509 INFO:teuthology.orchestra.run.smithi113.stdout:(116/147): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-01-16T13:49:19.569 INFO:teuthology.orchestra.run.smithi113.stdout:(117/147): zlib-devel-1.2.11-18.el8_5.x86_64.rp 749 kB/s | 58 kB 00:00 2024-01-16T13:49:19.595 INFO:teuthology.orchestra.run.smithi124.stdout:(48/147): python3-cheroot-8.5.2-1.el8.noarch.rp 291 kB/s | 173 kB 00:00 2024-01-16T13:49:19.601 INFO:teuthology.orchestra.run.smithi113.stdout:(118/147): libcom_err-devel-1.45.6-4.el8.x86_64 419 kB/s | 39 kB 00:00 2024-01-16T13:49:19.626 INFO:teuthology.orchestra.run.smithi113.stdout:(119/147): fuse-2.9.7-17.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2024-01-16T13:49:19.660 INFO:teuthology.orchestra.run.smithi113.stdout:(120/147): libgfortran-8.5.0-20.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2024-01-16T13:49:19.685 INFO:teuthology.orchestra.run.smithi113.stdout:(121/147): libquadmath-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-01-16T13:49:19.695 INFO:teuthology.orchestra.run.smithi124.stdout:(49/147): python3-google-auth-1.1.1-10.el8.noar 304 kB/s | 81 kB 00:00 2024-01-16T13:49:19.719 INFO:teuthology.orchestra.run.smithi113.stdout:(122/147): libuuid-devel-2.32.1-43.el8.x86_64.r 1.7 MB/s | 100 kB 00:00 2024-01-16T13:49:19.744 INFO:teuthology.orchestra.run.smithi113.stdout:(123/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2024-01-16T13:49:19.778 INFO:teuthology.orchestra.run.smithi113.stdout:(124/147): python3-cryptography-3.2.1-6.el8.x86 9.3 MB/s | 559 kB 00:00 2024-01-16T13:49:19.795 INFO:teuthology.orchestra.run.smithi113.stdout:(125/147): xfsprogs-devel-5.0.0-12.el8.x86_64.r 1.4 MB/s | 73 kB 00:00 2024-01-16T13:49:19.839 INFO:teuthology.orchestra.run.smithi113.stdout:(126/147): elfutils-libelf-devel-0.189-3.el8.x8 1.0 MB/s | 62 kB 00:00 2024-01-16T13:49:19.862 INFO:teuthology.orchestra.run.smithi124.stdout:(50/147): python3-jaraco-6.2-6.el8.noarch.rpm 42 kB/s | 11 kB 00:00 2024-01-16T13:49:19.862 INFO:teuthology.orchestra.run.smithi113.stdout:(127/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 701 kB/s | 47 kB 00:00 2024-01-16T13:49:19.904 INFO:teuthology.orchestra.run.smithi113.stdout:(128/147): krb5-devel-1.18.2-26.el8_9.x86_64.rp 8.5 MB/s | 562 kB 00:00 2024-01-16T13:49:19.945 INFO:teuthology.orchestra.run.smithi124.stdout:(51/147): python3-jaraco-functools-2.0-4.el8.no 70 kB/s | 18 kB 00:00 2024-01-16T13:49:19.955 INFO:teuthology.orchestra.run.smithi113.stdout:(129/147): NetworkManager-initscripts-updown-1. 2.8 MB/s | 144 kB 00:00 2024-01-16T13:49:20.013 INFO:teuthology.orchestra.run.smithi113.stdout:(130/147): openssl-devel-1.1.1k-12.el8_9.x86_64 15 MB/s | 2.3 MB 00:00 2024-01-16T13:49:20.189 INFO:teuthology.orchestra.run.smithi113.stdout:(131/147): python3-scipy-1.0.0-21.module+el8.9. 11 MB/s | 14 MB 00:01 2024-01-16T13:49:20.232 INFO:teuthology.orchestra.run.smithi124.stdout:(52/147): python3-logutils-0.3.5-11.el8.noarch. 170 kB/s | 49 kB 00:00 2024-01-16T13:49:20.247 INFO:teuthology.orchestra.run.smithi113.stdout:(132/147): elfutils-libelf-0.189-3.el8.x86_64.r 3.9 MB/s | 232 kB 00:00 2024-01-16T13:49:20.298 INFO:teuthology.orchestra.run.smithi113.stdout:(133/147): elfutils-libs-0.189-3.el8.x86_64.rpm 5.9 MB/s | 303 kB 00:00 2024-01-16T13:49:20.348 INFO:teuthology.orchestra.run.smithi113.stdout:(134/147): libmount-2.32.1-43.el8.x86_64.rpm 4.6 MB/s | 237 kB 00:00 2024-01-16T13:49:20.388 INFO:teuthology.orchestra.run.smithi124.stdout:(53/147): ceph-mgr-diskprediction-local-18.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-01-16T13:49:20.423 INFO:teuthology.orchestra.run.smithi113.stdout:(135/147): libsmartcols-2.32.1-43.el8.x86_64.rp 2.3 MB/s | 179 kB 00:00 2024-01-16T13:49:20.473 INFO:teuthology.orchestra.run.smithi113.stdout:(136/147): libuuid-2.32.1-43.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-01-16T13:49:20.496 INFO:teuthology.orchestra.run.smithi124.stdout:(54/147): python3-more-itertools-7.2.0-3.el8.no 222 kB/s | 59 kB 00:00 2024-01-16T13:49:20.565 INFO:teuthology.orchestra.run.smithi113.stdout:(137/147): librbd1-18.0.0-7629.g73a4532f.el8.x8 6.7 MB/s | 3.7 MB 00:00 2024-01-16T13:49:20.616 INFO:teuthology.orchestra.run.smithi113.stdout:(138/147): util-linux-user-2.32.1-43.el8.x86_64 2.0 MB/s | 103 kB 00:00 2024-01-16T13:49:20.758 INFO:teuthology.orchestra.run.smithi113.stdout:(139/147): util-linux-2.32.1-43.el8.x86_64.rpm 8.7 MB/s | 2.5 MB 00:00 2024-01-16T13:49:20.816 INFO:teuthology.orchestra.run.smithi113.stdout:(140/147): librados2-18.0.0-7629.g73a4532f.el8. 4.4 MB/s | 3.8 MB 00:00 2024-01-16T13:49:20.822 INFO:teuthology.orchestra.run.smithi124.stdout:(55/147): python3-kubernetes-11.0.0-6.el8.noarc 1.2 MB/s | 1.2 MB 00:00 2024-01-16T13:49:20.842 INFO:teuthology.orchestra.run.smithi113.stdout:(141/147): libblkid-2.32.1-43.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-01-16T13:49:20.867 INFO:teuthology.orchestra.run.smithi113.stdout:(142/147): libfdisk-2.32.1-43.el8.x86_64.rpm 4.9 MB/s | 254 kB 00:00 2024-01-16T13:49:20.909 INFO:teuthology.orchestra.run.smithi113.stdout:(143/147): iproute-6.2.0-5.el8_9.x86_64.rpm 2.8 MB/s | 854 kB 00:00 2024-01-16T13:49:20.947 INFO:teuthology.orchestra.run.smithi124.stdout:(56/147): python3-pecan-1.3.2-9.el8.noarch.rpm 628 kB/s | 283 kB 00:00 2024-01-16T13:49:20.970 INFO:teuthology.orchestra.run.smithi113.stdout:(144/147): openssl-1.1.1k-12.el8_9.x86_64.rpm 6.8 MB/s | 711 kB 00:00 2024-01-16T13:49:21.009 INFO:teuthology.orchestra.run.smithi113.stdout:(145/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 6.4 MB/s | 1.1 MB 00:00 2024-01-16T13:49:21.043 INFO:teuthology.orchestra.run.smithi113.stdout:(146/147): xfsdump-3.1.8-7.el8_9.x86_64.rpm 4.5 MB/s | 333 kB 00:00 2024-01-16T13:49:21.085 INFO:teuthology.orchestra.run.smithi113.stdout:(147/147): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.4 MB/s | 1.5 MB 00:00 2024-01-16T13:49:21.088 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-01-16T13:49:21.088 INFO:teuthology.orchestra.run.smithi113.stdout:Total 9.6 MB/s | 218 MB 00:22 2024-01-16T13:49:21.347 INFO:teuthology.orchestra.run.smithi124.stdout:(57/147): python3-portend-2.6-1.el8.noarch.rpm 30 kB/s | 16 kB 00:00 2024-01-16T13:49:21.389 INFO:teuthology.orchestra.run.smithi124.stdout:(58/147): python3-repoze-lru-0.7-6.el8.noarch.r 76 kB/s | 34 kB 00:00 2024-01-16T13:49:21.698 INFO:teuthology.orchestra.run.smithi124.stdout:(59/147): python3-natsort-7.1.1-2.el8.noarch.rp 46 kB/s | 60 kB 00:01 2024-01-16T13:49:21.873 INFO:teuthology.orchestra.run.smithi124.stdout:(60/147): python3-rsa-4.9-2.el8.noarch.rpm 133 kB/s | 64 kB 00:00 2024-01-16T13:49:21.898 INFO:teuthology.orchestra.run.smithi124.stdout:(61/147): python3-routes-2.4.1-12.el8.noarch.rp 357 kB/s | 196 kB 00:00 2024-01-16T13:49:21.990 INFO:teuthology.orchestra.run.smithi124.stdout:(62/147): python3-simplegeneric-0.8.1-17.el8.no 64 kB/s | 19 kB 00:00 2024-01-16T13:49:22.115 INFO:teuthology.orchestra.run.smithi124.stdout:(63/147): python3-singledispatch-3.4.0.3-18.el8 99 kB/s | 24 kB 00:00 2024-01-16T13:49:22.136 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-01-16T13:49:22.149 INFO:teuthology.orchestra.run.smithi124.stdout:(64/147): python3-tempora-1.14.1-5.el8.noarch.r 117 kB/s | 29 kB 00:00 2024-01-16T13:49:22.299 INFO:teuthology.orchestra.run.smithi124.stdout:(65/147): python3-trustme-0.6.0-4.el8.noarch.rp 89 kB/s | 27 kB 00:00 2024-01-16T13:49:22.433 INFO:teuthology.orchestra.run.smithi124.stdout:(66/147): python3-waitress-1.4.3-1.el8.noarch.r 805 kB/s | 255 kB 00:00 2024-01-16T13:49:22.450 INFO:teuthology.orchestra.run.smithi124.stdout:(67/147): python3-webob-1.8.5-1.el8.1.noarch.rp 836 kB/s | 251 kB 00:00 2024-01-16T13:49:22.533 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-01-16T13:49:22.533 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-01-16T13:49:22.675 INFO:teuthology.orchestra.run.smithi124.stdout:(68/147): python3-websocket-client-0.56.0-5.el8 163 kB/s | 61 kB 00:00 2024-01-16T13:49:22.700 INFO:teuthology.orchestra.run.smithi124.stdout:(69/147): python3-webtest-2.0.33-1.el8.noarch.r 320 kB/s | 85 kB 00:00 2024-01-16T13:49:22.717 INFO:teuthology.orchestra.run.smithi124.stdout:(70/147): python3-zc-lockfile-2.0-2.el8.noarch. 86 kB/s | 23 kB 00:00 2024-01-16T13:49:22.742 INFO:teuthology.orchestra.run.smithi124.stdout:(71/147): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2024-01-16T13:49:22.784 INFO:teuthology.orchestra.run.smithi124.stdout:(72/147): python3-jsonpointer-1.10-11.el8.noarc 300 kB/s | 20 kB 00:00 2024-01-16T13:49:22.802 INFO:teuthology.orchestra.run.smithi124.stdout:(73/147): python3-prettytable-0.7.2-14.el8.noar 832 kB/s | 44 kB 00:00 2024-01-16T13:49:22.818 INFO:teuthology.orchestra.run.smithi124.stdout:(74/147): python3-pyasn1-modules-0.3.7-6.el8.no 3.2 MB/s | 110 kB 00:00 2024-01-16T13:49:22.835 INFO:teuthology.orchestra.run.smithi124.stdout:(75/147): python3-jsonpatch-1.21-2.el8.noarch.r 794 kB/s | 27 kB 00:00 2024-01-16T13:49:22.860 INFO:teuthology.orchestra.run.smithi124.stdout:(76/147): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.9 MB/s | 126 kB 00:00 2024-01-16T13:49:22.894 INFO:teuthology.orchestra.run.smithi124.stdout:(77/147): python3-werkzeug-0.12.2-4.el8.noarch. 7.6 MB/s | 457 kB 00:00 2024-01-16T13:49:22.927 INFO:teuthology.orchestra.run.smithi124.stdout:(78/147): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2024-01-16T13:49:22.953 INFO:teuthology.orchestra.run.smithi124.stdout:(79/147): flex-2.6.1-9.el8.x86_64.rpm 5.3 MB/s | 320 kB 00:00 2024-01-16T13:49:22.986 INFO:teuthology.orchestra.run.smithi124.stdout:(80/147): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2024-01-16T13:49:23.012 INFO:teuthology.orchestra.run.smithi124.stdout:(81/147): python3-markupsafe-0.23-19.el8.x86_64 666 kB/s | 39 kB 00:00 2024-01-16T13:49:23.037 INFO:teuthology.orchestra.run.smithi124.stdout:(82/147): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2024-01-16T13:49:23.062 INFO:teuthology.orchestra.run.smithi124.stdout:(83/147): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-01-16T13:49:23.112 INFO:teuthology.orchestra.run.smithi124.stdout:(84/147): uuid-1.6.2-43.el8.x86_64.rpm 853 kB/s | 64 kB 00:00 2024-01-16T13:49:23.138 INFO:teuthology.orchestra.run.smithi124.stdout:(85/147): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.3 MB/s | 103 kB 00:00 2024-01-16T13:49:23.188 INFO:teuthology.orchestra.run.smithi124.stdout:(86/147): python3-jinja2-2.10.1-3.el8.noarch.rp 11 MB/s | 538 kB 00:00 2024-01-16T13:49:23.288 INFO:teuthology.orchestra.run.smithi124.stdout:(87/147): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2024-01-16T13:49:23.330 INFO:teuthology.orchestra.run.smithi124.stdout:(88/147): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2024-01-16T13:49:23.389 INFO:teuthology.orchestra.run.smithi124.stdout:(89/147): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2024-01-16T13:49:23.406 INFO:teuthology.orchestra.run.smithi124.stdout:(90/147): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.9 MB/s | 225 kB 00:00 2024-01-16T13:49:23.431 INFO:teuthology.orchestra.run.smithi124.stdout:(91/147): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2024-01-16T13:49:23.473 INFO:teuthology.orchestra.run.smithi124.stdout:(92/147): python3-mako-1.0.6-14.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2024-01-16T13:49:23.582 INFO:teuthology.orchestra.run.smithi124.stdout:(93/147): openblas-threads-0.3.15-6.el8.x86_64. 30 MB/s | 5.2 MB 00:00 2024-01-16T13:49:23.640 INFO:teuthology.orchestra.run.smithi124.stdout:(94/147): automake-1.16.1-8.el8.noarch.rpm 12 MB/s | 714 kB 00:00 2024-01-16T13:49:23.740 INFO:teuthology.orchestra.run.smithi124.stdout:(95/147): openblas-0.3.15-6.el8.x86_64.rpm 19 MB/s | 5.0 MB 00:00 2024-01-16T13:49:23.757 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-01-16T13:49:23.766 INFO:teuthology.orchestra.run.smithi124.stdout:(96/147): libpq-13.11-1.el8.x86_64.rpm 1.5 MB/s | 198 kB 00:00 2024-01-16T13:49:23.767 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-01-16T13:49:23.791 INFO:teuthology.orchestra.run.smithi124.stdout:(97/147): jq-1.6-7.el8.x86_64.rpm 3.9 MB/s | 202 kB 00:00 2024-01-16T13:49:23.817 INFO:teuthology.orchestra.run.smithi124.stdout:(98/147): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2024-01-16T13:49:23.875 INFO:teuthology.orchestra.run.smithi124.stdout:(99/147): postgresql-10.23-3.module+el8.9.0+208 18 MB/s | 1.5 MB 00:00 2024-01-16T13:49:23.908 INFO:teuthology.orchestra.run.smithi124.stdout:(100/147): postgresql-contrib-10.23-3.module+el 8.6 MB/s | 811 kB 00:00 2024-01-16T13:49:24.042 INFO:teuthology.orchestra.run.smithi124.stdout:(101/147): postgresql-server-10.23-3.module+el8 30 MB/s | 5.1 MB 00:00 2024-01-16T13:49:24.084 INFO:teuthology.orchestra.run.smithi124.stdout:(102/147): thrift-0.13.0-2.el8.x86_64.rpm 1.2 MB/s | 1.7 MB 00:01 2024-01-16T13:49:24.109 INFO:teuthology.orchestra.run.smithi124.stdout:(103/147): libverto-devel-0.3.0-5.el8.x86_64.rp 268 kB/s | 18 kB 00:00 2024-01-16T13:49:24.151 INFO:teuthology.orchestra.run.smithi124.stdout:(104/147): python3-jwt-1.6.1-2.el8.noarch.rpm 639 kB/s | 43 kB 00:00 2024-01-16T13:49:24.177 INFO:teuthology.orchestra.run.smithi124.stdout:(105/147): python3-oauthlib-2.1.0-1.el8.noarch. 2.5 MB/s | 155 kB 00:00 2024-01-16T13:49:24.202 INFO:teuthology.orchestra.run.smithi124.stdout:(106/147): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2024-01-16T13:49:24.219 INFO:teuthology.orchestra.run.smithi124.stdout:(107/147): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2024-01-16T13:49:24.268 INFO:teuthology.orchestra.run.smithi124.stdout:(108/147): libaio-devel-0.3.112-1.el8.x86_64.rp 285 kB/s | 19 kB 00:00 2024-01-16T13:49:24.286 INFO:teuthology.orchestra.run.smithi124.stdout:(109/147): libzstd-devel-1.4.4-1.el8.x86_64.rpm 656 kB/s | 44 kB 00:00 2024-01-16T13:49:24.319 INFO:teuthology.orchestra.run.smithi124.stdout:(110/147): pcre2-devel-10.32-2.el8.x86_64.rpm 11 MB/s | 605 kB 00:00 2024-01-16T13:49:24.345 INFO:teuthology.orchestra.run.smithi124.stdout:(111/147): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2024-01-16T13:49:24.362 INFO:teuthology.orchestra.run.smithi124.stdout:(112/147): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-01-16T13:49:24.387 INFO:teuthology.orchestra.run.smithi124.stdout:(113/147): libselinux-devel-2.9-5.el8.x86_64.rp 4.7 MB/s | 200 kB 00:00 2024-01-16T13:49:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:(114/147): keyutils-libs-devel-1.5.10-9.el8.x86 958 kB/s | 48 kB 00:00 2024-01-16T13:49:24.429 INFO:teuthology.orchestra.run.smithi124.stdout:(115/147): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-01-16T13:49:24.454 INFO:teuthology.orchestra.run.smithi124.stdout:(116/147): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.3 MB/s | 58 kB 00:00 2024-01-16T13:49:24.480 INFO:teuthology.orchestra.run.smithi124.stdout:(117/147): libcom_err-devel-1.45.6-4.el8.x86_64 762 kB/s | 39 kB 00:00 2024-01-16T13:49:24.505 INFO:teuthology.orchestra.run.smithi124.stdout:(118/147): fuse-2.9.7-17.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2024-01-16T13:49:24.539 INFO:teuthology.orchestra.run.smithi124.stdout:(119/147): libgfortran-8.5.0-20.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2024-01-16T13:49:24.556 INFO:teuthology.orchestra.run.smithi124.stdout:(120/147): libquadmath-8.5.0-20.el8.x86_64.rpm 3.4 MB/s | 172 kB 00:00 2024-01-16T13:49:24.581 INFO:teuthology.orchestra.run.smithi124.stdout:(121/147): libuuid-devel-2.32.1-43.el8.x86_64.r 2.3 MB/s | 100 kB 00:00 2024-01-16T13:49:24.606 INFO:teuthology.orchestra.run.smithi124.stdout:(122/147): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2024-01-16T13:49:24.640 INFO:teuthology.orchestra.run.smithi124.stdout:(123/147): python3-cryptography-3.2.1-6.el8.x86 9.3 MB/s | 559 kB 00:00 2024-01-16T13:49:24.665 INFO:teuthology.orchestra.run.smithi124.stdout:(124/147): xfsprogs-devel-5.0.0-12.el8.x86_64.r 1.2 MB/s | 73 kB 00:00 2024-01-16T13:49:24.690 INFO:teuthology.orchestra.run.smithi124.stdout:(125/147): elfutils-libelf-devel-0.189-3.el8.x8 1.2 MB/s | 62 kB 00:00 2024-01-16T13:49:24.716 INFO:teuthology.orchestra.run.smithi124.stdout:(126/147): librabbitmq-0.9.0-4.el8.x86_64.rpm 934 kB/s | 47 kB 00:00 2024-01-16T13:49:24.749 INFO:teuthology.orchestra.run.smithi124.stdout:(127/147): krb5-devel-1.18.2-26.el8_9.x86_64.rp 9.4 MB/s | 562 kB 00:00 2024-01-16T13:49:24.791 INFO:teuthology.orchestra.run.smithi124.stdout:(128/147): NetworkManager-initscripts-updown-1. 3.4 MB/s | 144 kB 00:00 2024-01-16T13:49:24.858 INFO:teuthology.orchestra.run.smithi124.stdout:(129/147): lua-devel-5.3.4-12.el8.x86_64.rpm 450 kB/s | 30 kB 00:00 2024-01-16T13:49:24.900 INFO:teuthology.orchestra.run.smithi124.stdout:(130/147): openssl-devel-1.1.1k-12.el8_9.x86_64 13 MB/s | 2.3 MB 00:00 2024-01-16T13:49:25.050 INFO:teuthology.orchestra.run.smithi124.stdout:(131/147): python3-scipy-1.0.0-21.module+el8.9. 12 MB/s | 14 MB 00:01 2024-01-16T13:49:25.092 INFO:teuthology.orchestra.run.smithi124.stdout:(132/147): elfutils-libelf-0.189-3.el8.x86_64.r 5.5 MB/s | 232 kB 00:00 2024-01-16T13:49:25.134 INFO:teuthology.orchestra.run.smithi124.stdout:(133/147): elfutils-libs-0.189-3.el8.x86_64.rpm 7.1 MB/s | 303 kB 00:00 2024-01-16T13:49:25.176 INFO:teuthology.orchestra.run.smithi124.stdout:(134/147): libmount-2.32.1-43.el8.x86_64.rpm 5.6 MB/s | 237 kB 00:00 2024-01-16T13:49:25.218 INFO:teuthology.orchestra.run.smithi124.stdout:(135/147): libsmartcols-2.32.1-43.el8.x86_64.rp 4.2 MB/s | 179 kB 00:00 2024-01-16T13:49:25.251 INFO:teuthology.orchestra.run.smithi124.stdout:(136/147): libuuid-2.32.1-43.el8.x86_64.rpm 2.9 MB/s | 99 kB 00:00 2024-01-16T13:49:25.327 INFO:teuthology.orchestra.run.smithi124.stdout:(137/147): util-linux-2.32.1-43.el8.x86_64.rpm 33 MB/s | 2.5 MB 00:00 2024-01-16T13:49:25.360 INFO:teuthology.orchestra.run.smithi124.stdout:(138/147): util-linux-user-2.32.1-43.el8.x86_64 3.0 MB/s | 103 kB 00:00 2024-01-16T13:49:25.436 INFO:teuthology.orchestra.run.smithi124.stdout:(139/147): iproute-6.2.0-5.el8_9.x86_64.rpm 11 MB/s | 854 kB 00:00 2024-01-16T13:49:25.478 INFO:teuthology.orchestra.run.smithi124.stdout:(140/147): libblkid-2.32.1-43.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-01-16T13:49:25.519 INFO:teuthology.orchestra.run.smithi124.stdout:(141/147): libfdisk-2.32.1-43.el8.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-01-16T13:49:25.578 INFO:teuthology.orchestra.run.smithi124.stdout:(142/147): xfsprogs-5.0.0-12.el8.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-01-16T13:49:25.620 INFO:teuthology.orchestra.run.smithi124.stdout:(143/147): openssl-1.1.1k-12.el8_9.x86_64.rpm 17 MB/s | 711 kB 00:00 2024-01-16T13:49:25.687 INFO:teuthology.orchestra.run.smithi124.stdout:(144/147): openssl-libs-1.1.1k-12.el8_9.x86_64. 22 MB/s | 1.5 MB 00:00 2024-01-16T13:49:25.729 INFO:teuthology.orchestra.run.smithi124.stdout:(145/147): xfsdump-3.1.8-7.el8_9.x86_64.rpm 7.8 MB/s | 333 kB 00:00 2024-01-16T13:49:26.654 INFO:teuthology.orchestra.run.smithi124.stdout:(146/147): librbd1-18.0.0-7629.g73a4532f.el8.x8 2.1 MB/s | 3.7 MB 00:01 2024-01-16T13:49:27.196 INFO:teuthology.orchestra.run.smithi124.stdout:(147/147): librados2-18.0.0-7629.g73a4532f.el8. 1.6 MB/s | 3.8 MB 00:02 2024-01-16T13:49:27.200 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2024-01-16T13:49:27.201 INFO:teuthology.orchestra.run.smithi124.stdout:Total 7.9 MB/s | 218 MB 00:27 2024-01-16T13:49:28.255 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2024-01-16T13:49:28.649 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2024-01-16T13:49:28.649 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2024-01-16T13:49:29.865 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2024-01-16T13:49:29.875 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2024-01-16T13:49:29.962 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-01-16T13:49:30.097 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-16T13:49:30.363 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-16T13:49:30.592 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-16T13:49:30.765 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/163 2024-01-16T13:49:30.895 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-16T13:49:31.011 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-16T13:49:31.187 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-16T13:49:31.289 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-16T13:49:31.551 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/163 2024-01-16T13:49:31.720 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/163 2024-01-16T13:49:31.822 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/163 2024-01-16T13:49:32.022 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/163 2024-01-16T13:49:32.245 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-16T13:49:32.649 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-16T13:49:32.837 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 9/163 2024-01-16T13:49:33.268 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 9/163 2024-01-16T13:49:33.470 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 10/163 2024-01-16T13:49:33.601 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 10/163 2024-01-16T13:49:33.787 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 11/163 2024-01-16T13:49:33.885 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 11/163 2024-01-16T13:49:34.678 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el8.x86_64 12/163 2024-01-16T13:49:34.836 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 13/163 2024-01-16T13:49:35.018 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 13/163 2024-01-16T13:49:35.269 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-16T13:49:35.358 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-16T13:49:35.427 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2024-01-16T13:49:35.534 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpq-13.11-1.el8.x86_64 15/163 2024-01-16T13:49:35.561 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-16T13:49:35.773 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 16/163 2024-01-16T13:49:35.831 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-16T13:49:35.989 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/163 2024-01-16T13:49:36.052 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/163 2024-01-16T13:49:36.205 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/163 2024-01-16T13:49:36.324 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 18/163 2024-01-16T13:49:36.398 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-16T13:49:36.463 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/163 2024-01-16T13:49:36.568 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 19/163 2024-01-16T13:49:36.648 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-16T13:49:36.708 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/163 2024-01-16T13:49:36.711 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 20/163 2024-01-16T13:49:36.854 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 20/163 2024-01-16T13:49:36.937 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/163 2024-01-16T13:49:36.978 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86 21/163 2024-01-16T13:49:37.080 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86 21/163 2024-01-16T13:49:37.106 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/163 2024-01-16T13:49:37.174 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/163 2024-01-16T13:49:37.235 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 22/163 2024-01-16T13:49:37.375 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/163 2024-01-16T13:49:37.400 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 22/163 2024-01-16T13:49:37.544 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 23/163 2024-01-16T13:49:37.598 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-16T13:49:38.009 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/163 2024-01-16T13:49:38.200 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 23/163 2024-01-16T13:49:38.214 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 9/163 2024-01-16T13:49:38.377 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 24/163 2024-01-16T13:49:38.627 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 9/163 2024-01-16T13:49:38.814 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 10/163 2024-01-16T13:49:38.944 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 10/163 2024-01-16T13:49:39.014 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 24/163 2024-01-16T13:49:39.089 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 11/163 2024-01-16T13:49:39.185 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 11/163 2024-01-16T13:49:39.227 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.15-6.el8.x86_64 25/163 2024-01-16T13:49:39.928 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 25/163 2024-01-16T13:49:39.972 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el8.x86_64 12/163 2024-01-16T13:49:40.114 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 13/163 2024-01-16T13:49:40.241 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 26/163 2024-01-16T13:49:40.295 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 13/163 2024-01-16T13:49:40.416 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 27/163 2024-01-16T13:49:40.547 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-16T13:49:40.580 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 28/163 2024-01-16T13:49:40.634 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/163 2024-01-16T13:49:40.803 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libpq-13.11-1.el8.x86_64 15/163 2024-01-16T13:49:40.888 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 29/163 2024-01-16T13:49:41.051 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 16/163 2024-01-16T13:49:41.202 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 30/163 2024-01-16T13:49:41.242 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/163 2024-01-16T13:49:41.464 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 31/163 2024-01-16T13:49:41.593 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 18/163 2024-01-16T13:49:41.641 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el8. 32/163 2024-01-16T13:49:41.796 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 19/163 2024-01-16T13:49:41.837 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 33/163 2024-01-16T13:49:41.939 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 20/163 2024-01-16T13:49:42.046 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 20/163 2024-01-16T13:49:42.138 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el8.x86_6 34/163 2024-01-16T13:49:42.214 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86 21/163 2024-01-16T13:49:42.289 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86 21/163 2024-01-16T13:49:42.489 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 22/163 2024-01-16T13:49:42.608 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 22/163 2024-01-16T13:49:42.780 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 23/163 2024-01-16T13:49:43.140 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 35/163 2024-01-16T13:49:43.407 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 23/163 2024-01-16T13:49:43.466 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 36/163 2024-01-16T13:49:43.572 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 24/163 2024-01-16T13:49:43.780 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 37/163 2024-01-16T13:49:43.976 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 38/163 2024-01-16T13:49:44.180 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 24/163 2024-01-16T13:49:44.364 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-0.3.15-6.el8.x86_64 25/163 2024-01-16T13:49:45.089 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 25/163 2024-01-16T13:49:45.361 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 26/163 2024-01-16T13:49:45.502 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 27/163 2024-01-16T13:49:45.717 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 28/163 2024-01-16T13:49:45.727 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 38/163 2024-01-16T13:49:46.014 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 29/163 2024-01-16T13:49:46.068 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 39/163 2024-01-16T13:49:46.295 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/163 2024-01-16T13:49:46.311 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 30/163 2024-01-16T13:49:46.478 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 41/163 2024-01-16T13:49:46.548 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 31/163 2024-01-16T13:49:46.685 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el8.x86_64 42/163 2024-01-16T13:49:46.736 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el8. 32/163 2024-01-16T13:49:46.857 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 33/163 2024-01-16T13:49:46.889 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el8.x86_64 43/163 2024-01-16T13:49:47.107 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el8.x86_6 44/163 2024-01-16T13:49:47.139 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el8.x86_6 34/163 2024-01-16T13:49:47.342 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/163 2024-01-16T13:49:47.427 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/163 2024-01-16T13:49:47.650 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-16T13:49:47.721 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-16T13:49:47.973 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/163 2024-01-16T13:49:48.122 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 35/163 2024-01-16T13:49:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 48/163 2024-01-16T13:49:48.452 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 36/163 2024-01-16T13:49:48.483 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-16T13:49:48.717 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 37/163 2024-01-16T13:49:48.921 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 38/163 2024-01-16T13:49:48.935 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-16T13:49:49.292 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-16T13:49:49.308 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-16T13:49:49.546 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.7-17.el8.x86_64 51/163 2024-01-16T13:49:49.721 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 52/163 2024-01-16T13:49:49.937 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/163 2024-01-16T13:49:50.170 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/163 2024-01-16T13:49:50.420 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/163 2024-01-16T13:49:50.654 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 38/163 2024-01-16T13:49:50.780 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 56/163 2024-01-16T13:49:51.013 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 39/163 2024-01-16T13:49:51.088 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 57/163 2024-01-16T13:49:51.241 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rsa-4.9-2.el8.noarch 40/163 2024-01-16T13:49:51.329 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/163 2024-01-16T13:49:51.449 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 41/163 2024-01-16T13:49:51.513 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 59/163 2024-01-16T13:49:51.689 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el8.x86_64 42/163 2024-01-16T13:49:51.719 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 60/163 2024-01-16T13:49:51.935 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el8.x86_64 43/163 2024-01-16T13:49:52.175 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 61/163 2024-01-16T13:49:52.279 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el8.x86_6 44/163 2024-01-16T13:49:52.518 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 62/163 2024-01-16T13:49:52.629 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 45/163 2024-01-16T13:49:52.707 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 45/163 2024-01-16T13:49:52.742 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 63/163 2024-01-16T13:49:52.920 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-16T13:49:52.991 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 46/163 2024-01-16T13:49:53.003 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 64/163 2024-01-16T13:49:53.217 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 47/163 2024-01-16T13:49:53.245 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 65/163 2024-01-16T13:49:53.436 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 48/163 2024-01-16T13:49:53.599 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 66/163 2024-01-16T13:49:53.679 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-16T13:49:53.823 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 67/163 2024-01-16T13:49:53.999 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 68/163 2024-01-16T13:49:54.108 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 49/163 2024-01-16T13:49:54.243 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 69/163 2024-01-16T13:49:54.412 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 70/163 2024-01-16T13:49:54.429 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-16T13:49:54.441 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 50/163 2024-01-16T13:49:54.596 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 51/163 2024-01-16T13:49:54.686 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 71/163 2024-01-16T13:49:54.833 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : fuse-2.9.7-17.el8.x86_64 52/163 2024-01-16T13:49:54.934 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 72/163 2024-01-16T13:49:55.026 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 53/163 2024-01-16T13:49:55.192 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 54/163 2024-01-16T13:49:55.203 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : automake-1.16.1-8.el8.noarch 73/163 2024-01-16T13:49:55.406 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 55/163 2024-01-16T13:49:55.407 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/163 2024-01-16T13:49:55.492 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/163 2024-01-16T13:49:55.615 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 56/163 2024-01-16T13:49:55.681 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 75/163 2024-01-16T13:49:55.765 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 75/163 2024-01-16T13:49:55.929 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 57/163 2024-01-16T13:49:55.990 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-7.el8.x86_64 76/163 2024-01-16T13:49:56.138 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 77/163 2024-01-16T13:49:56.224 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 58/163 2024-01-16T13:49:56.331 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 78/163 2024-01-16T13:49:56.417 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 59/163 2024-01-16T13:49:56.533 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 79/163 2024-01-16T13:49:56.607 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 60/163 2024-01-16T13:49:56.699 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 80/163 2024-01-16T13:49:56.813 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 61/163 2024-01-16T13:49:56.903 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 81/163 2024-01-16T13:49:57.183 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 62/163 2024-01-16T13:49:57.187 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 82/163 2024-01-16T13:49:57.421 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 83/163 2024-01-16T13:49:57.489 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 63/163 2024-01-16T13:49:57.627 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 84/163 2024-01-16T13:49:57.722 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 64/163 2024-01-16T13:49:57.809 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 85/163 2024-01-16T13:49:57.966 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 65/163 2024-01-16T13:49:58.040 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 86/163 2024-01-16T13:49:58.209 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 66/163 2024-01-16T13:49:58.277 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 87/163 2024-01-16T13:49:58.543 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 88/163 2024-01-16T13:49:58.555 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 67/163 2024-01-16T13:49:58.751 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 89/163 2024-01-16T13:49:58.761 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 68/163 2024-01-16T13:49:58.937 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 90/163 2024-01-16T13:49:58.945 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 69/163 2024-01-16T13:49:59.155 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 70/163 2024-01-16T13:49:59.189 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 91/163 2024-01-16T13:49:59.349 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 71/163 2024-01-16T13:49:59.450 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 92/163 2024-01-16T13:49:59.644 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 93/163 2024-01-16T13:49:59.671 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 72/163 2024-01-16T13:49:59.884 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-2.6-1.el8.noarch 94/163 2024-01-16T13:49:59.904 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : automake-1.16.1-8.el8.noarch 73/163 2024-01-16T13:50:00.096 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 95/163 2024-01-16T13:50:00.128 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/163 2024-01-16T13:50:00.208 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/163 2024-01-16T13:50:00.320 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 96/163 2024-01-16T13:50:00.370 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 75/163 2024-01-16T13:50:00.450 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 75/163 2024-01-16T13:50:00.580 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 97/163 2024-01-16T13:50:00.686 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : jq-1.6-7.el8.x86_64 76/163 2024-01-16T13:50:00.842 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 77/163 2024-01-16T13:50:01.010 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 78/163 2024-01-16T13:50:01.204 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 79/163 2024-01-16T13:50:01.361 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 80/163 2024-01-16T13:50:01.522 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 81/163 2024-01-16T13:50:01.675 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 98/163 2024-01-16T13:50:01.777 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 82/163 2024-01-16T13:50:01.967 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 83/163 2024-01-16T13:50:02.017 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 99/163 2024-01-16T13:50:02.148 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 84/163 2024-01-16T13:50:02.363 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 85/163 2024-01-16T13:50:02.577 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 86/163 2024-01-16T13:50:02.604 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 100/163 2024-01-16T13:50:02.788 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 87/163 2024-01-16T13:50:03.024 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 88/163 2024-01-16T13:50:03.198 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 89/163 2024-01-16T13:50:03.299 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 101/163 2024-01-16T13:50:03.384 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 90/163 2024-01-16T13:50:03.629 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 91/163 2024-01-16T13:50:03.658 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 102/163 2024-01-16T13:50:03.822 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 92/163 2024-01-16T13:50:03.947 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : luarocks-3.5.0-1.el8.noarch 103/163 2024-01-16T13:50:04.016 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 93/163 2024-01-16T13:50:04.179 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 104/163 2024-01-16T13:50:04.277 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-portend-2.6-1.el8.noarch 94/163 2024-01-16T13:50:04.422 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 105/163 2024-01-16T13:50:04.502 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 95/163 2024-01-16T13:50:04.700 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 96/163 2024-01-16T13:50:04.891 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 97/163 2024-01-16T13:50:05.923 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 98/163 2024-01-16T13:50:06.214 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 99/163 2024-01-16T13:50:06.434 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 100/163 2024-01-16T13:50:06.751 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 101/163 2024-01-16T13:50:07.088 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 102/163 2024-01-16T13:50:07.344 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : luarocks-3.5.0-1.el8.noarch 103/163 2024-01-16T13:50:07.568 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 104/163 2024-01-16T13:50:07.761 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 105/163 2024-01-16T13:50:08.271 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 106/163 2024-01-16T13:50:10.053 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 106/163 2024-01-16T13:50:10.633 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 106/163 2024-01-16T13:50:10.938 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 107/163 2024-01-16T13:50:11.023 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 107/163 2024-01-16T13:50:11.228 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 108/163 2024-01-16T13:50:11.461 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 106/163 2024-01-16T13:50:11.694 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 106/163 2024-01-16T13:50:12.268 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 106/163 2024-01-16T13:50:12.552 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 107/163 2024-01-16T13:50:12.630 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 107/163 2024-01-16T13:50:12.831 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 108/163 2024-01-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 108/163 2024-01-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-01-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-01-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /run 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-01-16T13:50:30.472 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:50:30.824 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 109/163 2024-01-16T13:50:32.422 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 109/163 2024-01-16T13:50:32.658 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 110/163 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 108/163 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /sys 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /proc 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /dev 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /run 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /mnt 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /var/tmp 2024-01-16T13:50:32.719 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /home 2024-01-16T13:50:32.720 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /tmp 2024-01-16T13:50:32.720 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /dev 2024-01-16T13:50:32.720 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:50:32.880 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 110/163 2024-01-16T13:50:33.048 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 111/163 2024-01-16T13:50:33.116 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 109/163 2024-01-16T13:50:33.509 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 111/163 2024-01-16T13:50:33.715 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 112/163 2024-01-16T13:50:33.733 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 112/163 2024-01-16T13:50:34.795 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 109/163 2024-01-16T13:50:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 110/163 2024-01-16T13:50:35.204 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 110/163 2024-01-16T13:50:35.365 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 111/163 2024-01-16T13:50:35.609 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 113/163 2024-01-16T13:50:35.834 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 113/163 2024-01-16T13:50:35.846 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 111/163 2024-01-16T13:50:35.980 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 114/163 2024-01-16T13:50:35.999 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 112/163 2024-01-16T13:50:36.016 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 112/163 2024-01-16T13:50:36.124 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 115/163 2024-01-16T13:50:36.372 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noa 116/163 2024-01-16T13:50:36.593 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noa 116/163 2024-01-16T13:50:36.873 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.n 117/163 2024-01-16T13:50:37.818 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 113/163 2024-01-16T13:50:37.844 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.n 117/163 2024-01-16T13:50:37.993 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 113/163 2024-01-16T13:50:38.039 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 118/163 2024-01-16T13:50:38.133 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 118/163 2024-01-16T13:50:38.147 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 114/163 2024-01-16T13:50:38.316 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 115/163 2024-01-16T13:50:38.508 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 119/163 2024-01-16T13:50:38.531 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noa 116/163 2024-01-16T13:50:38.748 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 120/163 2024-01-16T13:50:38.758 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noa 116/163 2024-01-16T13:50:38.787 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 120/163 2024-01-16T13:50:38.947 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 121/163 2024-01-16T13:50:38.959 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 121/163 2024-01-16T13:50:39.048 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.n 117/163 2024-01-16T13:50:40.015 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.n 117/163 2024-01-16T13:50:40.164 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 118/163 2024-01-16T13:50:40.259 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 118/163 2024-01-16T13:50:40.314 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el8.x86_64 122/163 2024-01-16T13:50:40.523 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 123/163 2024-01-16T13:50:40.558 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 123/163 2024-01-16T13:50:40.634 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 119/163 2024-01-16T13:50:40.747 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 124/163 2024-01-16T13:50:40.790 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 124/163 2024-01-16T13:50:40.823 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 120/163 2024-01-16T13:50:40.861 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 120/163 2024-01-16T13:50:40.971 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 125/163 2024-01-16T13:50:41.031 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 121/163 2024-01-16T13:50:41.043 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 121/163 2024-01-16T13:50:41.330 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 125/163 2024-01-16T13:50:41.546 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 126/163 2024-01-16T13:50:42.360 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el8.x86_64 122/163 2024-01-16T13:50:42.523 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 123/163 2024-01-16T13:50:42.558 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 123/163 2024-01-16T13:50:42.731 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 124/163 2024-01-16T13:50:42.774 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 124/163 2024-01-16T13:50:42.939 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 125/163 2024-01-16T13:50:43.297 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 125/163 2024-01-16T13:50:43.489 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 126/163 2024-01-16T13:50:46.227 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 126/163 2024-01-16T13:50:46.722 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el8.x86_64 127/163 2024-01-16T13:50:47.047 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 128/163 2024-01-16T13:50:47.270 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libtool-2.4.6-25.el8.x86_64 129/163 2024-01-16T13:50:47.959 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 129/163 2024-01-16T13:50:48.152 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 126/163 2024-01-16T13:50:48.291 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/163 2024-01-16T13:50:48.608 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 131/163 2024-01-16T13:50:48.624 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el8.x86_64 127/163 2024-01-16T13:50:48.809 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 132/163 2024-01-16T13:50:48.948 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 128/163 2024-01-16T13:50:49.056 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/163 2024-01-16T13:50:49.172 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libtool-2.4.6-25.el8.x86_64 129/163 2024-01-16T13:50:49.249 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/163 2024-01-16T13:50:49.454 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el8.x86_ 135/163 2024-01-16T13:50:49.861 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 129/163 2024-01-16T13:50:50.185 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/163 2024-01-16T13:50:50.469 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 131/163 2024-01-16T13:50:50.652 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 132/163 2024-01-16T13:50:50.907 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 133/163 2024-01-16T13:50:51.092 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 134/163 2024-01-16T13:50:51.272 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el8.x86_ 135/163 2024-01-16T13:50:51.712 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-16T13:50:52.011 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-16T13:50:52.049 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-16T13:50:52.524 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 137/163 2024-01-16T13:50:52.901 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 138/163 2024-01-16T13:50:53.156 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 139/163 2024-01-16T13:50:53.369 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el8.noarch 140/163 2024-01-16T13:50:53.527 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-16T13:50:53.561 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 141/163 2024-01-16T13:50:53.838 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-16T13:50:53.876 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 136/163 2024-01-16T13:50:53.881 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el8.x86_64 142/163 2024-01-16T13:50:54.145 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 143/163 2024-01-16T13:50:54.302 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 144/163 2024-01-16T13:50:54.312 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 144/163 2024-01-16T13:50:54.334 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 137/163 2024-01-16T13:50:54.498 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 145/163 2024-01-16T13:50:54.661 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 138/163 2024-01-16T13:50:54.726 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flex-2.6.1-9.el8.x86_64 146/163 2024-01-16T13:50:54.807 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 146/163 2024-01-16T13:50:54.917 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 139/163 2024-01-16T13:50:55.043 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : bison-3.0.4-10.el8.x86_64 147/163 2024-01-16T13:50:55.057 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 147/163 2024-01-16T13:50:55.104 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el8.noarch 140/163 2024-01-16T13:50:55.207 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 148/163 2024-01-16T13:50:55.313 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 141/163 2024-01-16T13:50:55.429 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 149/163 2024-01-16T13:50:55.589 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/163 2024-01-16T13:50:55.589 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-16T13:50:55.623 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el8.x86_64 142/163 2024-01-16T13:50:55.839 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 143/163 2024-01-16T13:50:55.888 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-16T13:50:55.888 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-16T13:50:56.038 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 144/163 2024-01-16T13:50:56.048 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 144/163 2024-01-16T13:50:56.122 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-16T13:50:56.122 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-16T13:50:56.216 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 145/163 2024-01-16T13:50:56.383 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-16T13:50:56.420 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : flex-2.6.1-9.el8.x86_64 146/163 2024-01-16T13:50:56.500 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 146/163 2024-01-16T13:50:56.595 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 154/163 2024-01-16T13:50:56.712 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : bison-3.0.4-10.el8.x86_64 147/163 2024-01-16T13:50:56.726 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 147/163 2024-01-16T13:50:56.795 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 155/163 2024-01-16T13:50:56.883 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 148/163 2024-01-16T13:50:57.028 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 156/163 2024-01-16T13:50:57.028 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-16T13:50:57.089 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 149/163 2024-01-16T13:50:57.216 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 150/163 2024-01-16T13:50:57.216 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-16T13:50:57.279 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-16T13:50:57.279 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 158/163 2024-01-16T13:50:57.474 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 151/163 2024-01-16T13:50:57.474 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-16T13:50:57.520 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 158/163 2024-01-16T13:50:57.520 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-16T13:50:57.724 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 152/163 2024-01-16T13:50:57.724 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-16T13:50:57.762 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-16T13:50:57.762 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-16T13:50:57.977 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 153/163 2024-01-16T13:50:57.986 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-16T13:50:57.987 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-16T13:50:58.164 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 154/163 2024-01-16T13:50:58.237 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-16T13:50:58.354 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 155/163 2024-01-16T13:50:58.428 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 162/163 2024-01-16T13:50:58.429 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-16T13:50:58.573 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 156/163 2024-01-16T13:50:58.573 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-16T13:50:58.831 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 157/163 2024-01-16T13:50:58.831 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 158/163 2024-01-16T13:50:59.081 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 158/163 2024-01-16T13:50:59.081 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-16T13:50:59.306 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 159/163 2024-01-16T13:50:59.306 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el8.x86_64 1/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 2/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 3/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 4/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 6/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 7/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 8/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 9/163 2024-01-16T13:50:59.354 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 10/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 11/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el8.x86_64 12/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 13/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el8.x86_ 14/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 15/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 16/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el8.x86_6 17/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86 18/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 19/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el8. 21/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el8.x86_6 22/163 2024-01-16T13:50:59.355 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el8.x86_64 23/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el8.x86_64 24/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el8.x86_64 25/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 26/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 27/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el8.x86_64 28/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/163 2024-01-16T13:50:59.356 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noa 30/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.n 31/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 34/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 36/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 37/163 2024-01-16T13:50:59.369 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el8.noarch 38/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 42/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 43/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 44/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 45/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 46/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 47/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 48/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 49/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/163 2024-01-16T13:50:59.370 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 56/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 57/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-2.6-1.el8.noarch 58/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/163 2024-01-16T13:50:59.371 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 71/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 72/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 73/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 74/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 75/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 76/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 77/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 78/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : bison-3.0.4-10.el8.x86_64 79/163 2024-01-16T13:50:59.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flex-2.6.1-9.el8.x86_64 80/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 81/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 82/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 83/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 84/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 85/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 86/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 87/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 88/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 89/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 90/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 91/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 92/163 2024-01-16T13:50:59.373 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 93/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 94/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 95/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : automake-1.16.1-8.el8.noarch 96/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpq-13.11-1.el8.x86_64 97/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-7.el8.x86_64 98/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 99/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 100/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 101/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 102/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 103/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 104/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 106/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 107/163 2024-01-16T13:50:59.374 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 108/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 110/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 111/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 112/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 113/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 114/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 115/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 116/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 117/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 118/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 119/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 120/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 121/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 122/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 123/163 2024-01-16T13:50:59.375 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 124/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 125/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 126/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 127/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 128/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 129/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 130/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 131/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 132/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 133/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 134/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 135/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 136/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 137/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 138/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 139/163 2024-01-16T13:50:59.376 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 140/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 141/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 142/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 143/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 144/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 145/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 146/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 147/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 148/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 149/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 150/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 151/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 152/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 153/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 154/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 155/163 2024-01-16T13:50:59.377 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 156/163 2024-01-16T13:50:59.378 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 157/163 2024-01-16T13:50:59.378 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 158/163 2024-01-16T13:50:59.378 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 159/163 2024-01-16T13:50:59.378 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 160/163 2024-01-16T13:50:59.378 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/163 2024-01-16T13:50:59.378 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 162/163 2024-01-16T13:50:59.573 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 160/163 2024-01-16T13:50:59.573 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-16T13:50:59.806 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 161/163 2024-01-16T13:50:59.998 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 162/163 2024-01-16T13:50:59.998 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 163/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el8.x86_64 1/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 2/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 3/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 4/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 6/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 7/163 2024-01-16T13:51:00.869 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 8/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 9/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 10/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 11/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el8.x86_64 12/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 13/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el8.x86_ 14/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 15/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 16/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el8.x86_6 17/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86 18/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 19/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/163 2024-01-16T13:51:00.870 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el8. 21/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el8.x86_6 22/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el8.x86_64 23/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el8.x86_64 24/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el8.x86_64 25/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 26/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 27/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el8.x86_64 28/163 2024-01-16T13:51:00.871 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noa 30/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.n 31/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 34/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 36/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 37/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el8.noarch 38/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/163 2024-01-16T13:51:00.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 42/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 43/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 44/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 45/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 46/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 47/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 48/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 49/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 50/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 51/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 52/163 2024-01-16T13:51:00.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 53/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 54/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 55/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 56/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 57/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-portend-2.6-1.el8.noarch 58/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 59/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 60/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 61/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 62/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 63/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 64/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 65/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 66/163 2024-01-16T13:51:00.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 67/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 68/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 69/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 70/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 71/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 72/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 73/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 74/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 75/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 76/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 77/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 78/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : bison-3.0.4-10.el8.x86_64 79/163 2024-01-16T13:51:00.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : flex-2.6.1-9.el8.x86_64 80/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 81/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 82/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 83/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 84/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 85/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 86/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 87/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 88/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 89/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 90/163 2024-01-16T13:51:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 91/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 92/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 93/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 94/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 95/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : automake-1.16.1-8.el8.noarch 96/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libpq-13.11-1.el8.x86_64 97/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : jq-1.6-7.el8.x86_64 98/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 99/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 100/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 101/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 102/163 2024-01-16T13:51:00.890 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 103/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/163 2024-01-16T13:51:00.891 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 122/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 123/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 124/163 2024-01-16T13:51:00.892 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 125/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 126/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 127/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 128/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 130/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 131/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 132/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 133/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 134/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 135/163 2024-01-16T13:51:00.893 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 136/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 137/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 138/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 139/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 140/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 141/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 142/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 143/163 2024-01-16T13:51:00.894 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 144/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 145/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 146/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 147/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 148/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 149/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 150/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 151/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 152/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 153/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 154/163 2024-01-16T13:51:00.895 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 155/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 156/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 157/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 158/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 159/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 160/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/163 2024-01-16T13:51:00.896 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 162/163 2024-01-16T13:51:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/163 2024-01-16T13:51:05.755 INFO:teuthology.orchestra.run.smithi113.stdout:Installed products updated. 2024-01-16T13:51:07.153 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.154 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: automake-1.16.1-8.el8.noarch 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: bison-3.0.4-10.el8.x86_64 2024-01-16T13:51:07.155 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.156 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-16T13:51:07.157 INFO:teuthology.orchestra.run.smithi113.stdout: flex-2.6.1-9.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-7.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T13:51:07.158 INFO:teuthology.orchestra.run.smithi113.stdout: libpq-13.11-1.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-16T13:51:07.159 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks-3.5.0-1.el8.noarch 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-16T13:51:07.160 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T13:51:07.161 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T13:51:07.162 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T13:51:07.163 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.164 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:07.165 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-01-16T13:51:07.167 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 163/163 2024-01-16T13:51:07.167 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2024-01-16T13:51:07.375 DEBUG:teuthology.parallel:result is None 2024-01-16T13:51:08.303 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:08.303 INFO:teuthology.orchestra.run.smithi124.stdout:Upgraded: 2024-01-16T13:51:08.303 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-16T13:51:08.303 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-16T13:51:08.303 INFO:teuthology.orchestra.run.smithi124.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-16T13:51:08.303 INFO:teuthology.orchestra.run.smithi124.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: librados2-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-16T13:51:08.304 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: automake-1.16.1-8.el8.noarch 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: bison-3.0.4-10.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.305 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el8.noarch 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: flex-2.6.1-9.el8.x86_64 2024-01-16T13:51:08.306 INFO:teuthology.orchestra.run.smithi124.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: jq-1.6-7.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libpq-13.11-1.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-16T13:51:08.307 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-01-16T13:51:08.308 INFO:teuthology.orchestra.run.smithi124.stdout: luarocks-3.5.0-1.el8.noarch 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.309 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T13:51:08.310 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T13:51:08.311 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T13:51:08.312 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-16T13:51:08.313 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T13:51:08.314 INFO:teuthology.orchestra.run.smithi124.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-16T13:51:08.314 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:08.314 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2024-01-16T13:51:08.528 DEBUG:teuthology.parallel:result is None 2024-01-16T13:51:08.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T13:51:08.755 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T13:51:08.783 INFO:teuthology.orchestra.run.smithi113.stdout:18.0.0-7629.g73a4532f.el8 2024-01-16T13:51:08.784 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el8 2024-01-16T13:51:08.784 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T13:51:08.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T13:51:09.005 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T13:51:09.029 INFO:teuthology.orchestra.run.smithi124.stdout:18.0.0-7629.g73a4532f.el8 2024-01-16T13:51:09.029 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el8 2024-01-16T13:51:09.029 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T13:51:09.031 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T13:51:09.031 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:09.031 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T13:51:09.079 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:09.079 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T13:51:09.122 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T13:51:09.122 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:09.122 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T13:51:09.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T13:51:09.220 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:09.220 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T13:51:09.248 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T13:51:09.316 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T13:51:09.316 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:09.316 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T13:51:09.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T13:51:09.410 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:09.411 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T13:51:09.437 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T13:51:09.503 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T13:51:09.503 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:09.503 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T13:51:09.530 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T13:51:09.596 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:09.596 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T13:51:09.622 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T13:51:09.689 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T13:51:09.767 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T13:51:09.767 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /var/log/ceph 2024-01-16T13:51:09.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 777 /var/log/ceph 2024-01-16T13:51:09.795 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T13:51:09.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T13:51:09.836 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T13:51:09.862 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T13:51:09.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T13:51:09.903 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T13:51:09.935 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-16T13:51:09.935 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-16T13:51:09.935 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-16T13:51:09.935 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:09.935 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T13:51:09.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T13:51:10.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T13:51:10.123 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:10.124 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T13:51:10.141 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T13:51:10.256 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T13:51:10.287 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T13:51:10.287 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T13:51:10.287 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7518127', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1988, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi113.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMgWGcqZsqMrhLUrN+rzIKJxjgom5ezT+W4YHAhRRit0V/oXcrcFOoaNVAfVwWQB1r4CmW4qRPPv3PnYCrSBGEE=', 'smithi124.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGv3mk9zPlpUpN698e9yQhF8X8unVOZl5D0AVavTlbH00i8d2sN7KnsA82RC5i+0EbqLMIENWV7ImBDF2RWbi4Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-16_13:34:04', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T13:51:10.288 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T13:51:10.290 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T13:51:10.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T13:51:10.347 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T13:51:10.406 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:10.406 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T13:51:10.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T13:51:10.430 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 170934 Links: 1 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-16 13:50:59.016813812 +0000 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-16 13:47:52.871788654 +0000 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-16 13:47:52.871788654 +0000 2024-01-16T13:51:10.487 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-16T13:51:10.488 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T13:51:10.551 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-16T13:51:10.552 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-16T13:51:10.552 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000220577 s, 2.3 MB/s 2024-01-16T13:51:10.553 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T13:51:10.611 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 177227 Links: 1 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-16 13:50:59.016813812 +0000 2024-01-16T13:51:10.668 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-16 13:47:53.207781603 +0000 2024-01-16T13:51:10.669 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-16 13:47:53.207781603 +0000 2024-01-16T13:51:10.669 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-16T13:51:10.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T13:51:10.733 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-16T13:51:10.733 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-16T13:51:10.733 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000296383 s, 1.7 MB/s 2024-01-16T13:51:10.735 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T13:51:10.792 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 174794 Links: 1 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-16 13:50:59.016813812 +0000 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-16 13:47:53.555774301 +0000 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-16 13:47:53.555774301 +0000 2024-01-16T13:51:10.849 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-16T13:51:10.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T13:51:10.914 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-16T13:51:10.914 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-16T13:51:10.914 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000194345 s, 2.6 MB/s 2024-01-16T13:51:10.915 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T13:51:10.972 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-01-16T13:51:11.029 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 180246 Links: 1 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-01-16 13:50:59.016813812 +0000 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-01-16 13:47:53.939766243 +0000 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-01-16 13:47:53.939766243 +0000 2024-01-16T13:51:11.030 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-01-16T13:51:11.030 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T13:51:11.094 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-01-16T13:51:11.094 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-01-16T13:51:11.094 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000233446 s, 2.2 MB/s 2024-01-16T13:51:11.095 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T13:51:11.153 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:11.153 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:11.153 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T13:51:11.172 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T13:51:11.172 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_1 2024-01-16T13:51:11.230 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T13:51:11.230 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:11.230 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 177295 Links: 1 2024-01-16T13:51:11.231 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:11.231 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:11.231 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2024-01-16 13:51:00.514542819 +0000 2024-01-16T13:51:11.231 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2024-01-16 13:47:48.124749017 +0000 2024-01-16T13:51:11.231 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2024-01-16 13:47:48.124749017 +0000 2024-01-16T13:51:11.231 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2024-01-16T13:51:11.231 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T13:51:11.297 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2024-01-16T13:51:11.298 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2024-01-16T13:51:11.298 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000267465 s, 1.9 MB/s 2024-01-16T13:51:11.299 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T13:51:11.357 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_2 2024-01-16T13:51:11.414 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T13:51:11.414 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:11.414 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 174863 Links: 1 2024-01-16T13:51:11.414 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2024-01-16 13:51:00.514542819 +0000 2024-01-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2024-01-16 13:47:48.473742970 +0000 2024-01-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2024-01-16 13:47:48.473742970 +0000 2024-01-16T13:51:11.415 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2024-01-16T13:51:11.415 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T13:51:11.480 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2024-01-16T13:51:11.480 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2024-01-16T13:51:11.480 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.00023786 s, 2.2 MB/s 2024-01-16T13:51:11.482 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T13:51:11.539 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_3 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 178355 Links: 1 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2024-01-16 13:51:00.514542819 +0000 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2024-01-16 13:47:48.818736992 +0000 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2024-01-16 13:47:48.818736992 +0000 2024-01-16T13:51:11.597 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2024-01-16T13:51:11.597 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T13:51:11.662 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2024-01-16T13:51:11.662 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2024-01-16T13:51:11.662 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000172657 s, 3.0 MB/s 2024-01-16T13:51:11.663 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T13:51:11.720 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_4 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 173003 Links: 1 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2024-01-16 13:51:00.514542819 +0000 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2024-01-16 13:47:49.169730909 +0000 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2024-01-16 13:47:49.169730909 +0000 2024-01-16T13:51:11.778 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2024-01-16T13:51:11.779 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T13:51:11.842 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2024-01-16T13:51:11.843 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2024-01-16T13:51:11.843 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000231828 s, 2.2 MB/s 2024-01-16T13:51:11.844 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T13:51:11.901 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:11.901 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T13:51:11.901 INFO:tasks.ceph:Generating config... 2024-01-16T13:51:11.902 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-16T13:51:11.903 INFO:tasks.ceph:[client] debug client = 20 2024-01-16T13:51:11.903 INFO:tasks.ceph:[client] debug ms = 1 2024-01-16T13:51:11.903 INFO:tasks.ceph:[client] pid file = 2024-01-16T13:51:11.903 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-16T13:51:11.903 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-16T13:51:11.903 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-16T13:51:11.903 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T13:51:11.904 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-16T13:51:11.904 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-16T13:51:11.904 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-16T13:51:11.904 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-16T13:51:11.904 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-16T13:51:11.905 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-16T13:51:11.905 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T13:51:11.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T13:51:11.974 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T13:51:11.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T13:51:12.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T13:51:12.051 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.113'), ('mon.b', '172.21.15.124'), ('mon.c', '[v2:172.21.15.124:3301,v1:172.21.15.124:6790]')] 2024-01-16T13:51:12.052 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.113,172.21.15.124,[v2:172.21.15.124:3301,v1:172.21.15.124:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-16T13:51:12.053 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:12.053 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T13:51:12.109 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.113 --add b 172.21.15.124 --addv c '[v2:172.21.15.124:3301,v1:172.21.15.124:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:monmaptool: generated fsid e77ab909-7680-4e53-a08d-a26ae42f1a8c 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:setting min_mon_release = pacific 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 0 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:fsid e77ab909-7680-4e53-a08d-a26ae42f1a8c 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:last_changed 2024-01-16T13:51:12.215708+0000 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-01-16T13:51:12.215708+0000 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:min_mon_release 16 (pacific) 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:election_strategy: 1 2024-01-16T13:51:12.197 INFO:teuthology.orchestra.run.smithi113.stdout:0: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.a 2024-01-16T13:51:12.198 INFO:teuthology.orchestra.run.smithi113.stdout:1: [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] mon.b 2024-01-16T13:51:12.198 INFO:teuthology.orchestra.run.smithi113.stdout:2: [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] mon.c 2024-01-16T13:51:12.198 INFO:teuthology.orchestra.run.smithi113.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-16T13:51:12.199 DEBUG:teuthology.orchestra.run.smithi113:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T13:51:12.215 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e77ab909-7680-4e53-a08d-a26ae42f1a8c... 2024-01-16T13:51:12.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T13:51:12.257 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T13:51:12.300 INFO:teuthology.orchestra.run.smithi113.stdout:[global] 2024-01-16T13:51:12.300 INFO:teuthology.orchestra.run.smithi113.stdout: chdir = "" 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: auth supported = cephx 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: filestore xattr use omap = true 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: mon clock drift allowed = 1.000 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: osd crush chooseleaf type = 0 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: auth debug = true 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on old message = true 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on bug = true 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.301 INFO:teuthology.orchestra.run.smithi113.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon pg warn max object skew = 0 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default size = 2 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow primary affinity = true 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow pg remap = true 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on legacy crush tunables = false 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on crush straw calc version zero = false 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on no sortbitwise = false 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on osd down out interval zero = false 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on too few osds = false 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T13:51:12.302 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: mon_allow_pool_size_one = true 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: osd default data pool replay window = 5 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: mon allow pool delete = true 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: mon cluster log file level = debug 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: debug asserts on shutdown = true 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: mon health detail to clog = false 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: mon host = "172.21.15.113,172.21.15.124,[v2:172.21.15.124:3301,v1:172.21.15.124:6790]" 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on skipped message = False 2024-01-16T13:51:12.303 INFO:teuthology.orchestra.run.smithi113.stdout: fsid = e77ab909-7680-4e53-a08d-a26ae42f1a8c 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout:[osd] 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd journal size = 100 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub load threshold = 5.0 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub max interval = 600 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd mclock profile = high_recovery_ops 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd recover clone overlap = true 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd recovery max chunk = 1048576 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug shutdown = true 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug op order = true 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug verify stray on activate = true 2024-01-16T13:51:12.304 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug trim objects = true 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: osd open classes on start = true 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug pg log writeout = true 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: osd deep scrub update digest min age = 30 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: osd map max advance = 10 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: journal zero on create = true 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: filestore ondisk finisher threads = 3 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: filestore apply finisher threads = 3 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: bdev debug aio = true 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug misdirected ops = true 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: bdev async discard = True 2024-01-16T13:51:12.305 INFO:teuthology.orchestra.run.smithi113.stdout: bdev enable discard = True 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore allocator = bitmap 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore block size = 96636764160 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore fsck on mount = True 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluefs = 1/20 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluestore = 1/20 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: debug osd = 20 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: debug rocksdb = 4/10 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd full ratio = 0.9 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd nearfull ratio = 0.8 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: osd failsafe full ratio = 0.95 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: osd objectstore = bluestore 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.306 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr] 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug mgr = 20 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: mgr/telemetry/nag = false 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug finisher = 20 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout:[mon] 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2024-01-16T13:51:12.307 INFO:teuthology.orchestra.run.smithi113.stdout: debug paxos = 20 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: mon data avail warn = 5 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: mon mgr mkfs grace = 240 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd reporter subtree level = osd 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd prime pg temp = true 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.308 INFO:teuthology.orchestra.run.smithi113.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: # 1m isn't quite enough 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: mon_down_mkfs_grace = 2m 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_filestore_osds = false 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: mon op complaint time = 120 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout:[client] 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: rgw cache enabled = true 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable ops log = true 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable usage log = true 2024-01-16T13:51:12.309 INFO:teuthology.orchestra.run.smithi113.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: client mount timeout = 600 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = "" 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.a] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.b] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.c] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout:[mds] 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds = 20 2024-01-16T13:51:12.310 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds balancer = 20 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug frag = True 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug scatterstat = True 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: mds op complaint time = 180 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: mds verify scatter = True 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2024-01-16T13:51:12.311 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2024-01-16T13:51:12.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: mon_allow_pool_size_one = true 2024-01-16T13:51:12.313 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = "172.21.15.113,172.21.15.124,[v2:172.21.15.124:3301,v1:172.21.15.124:6790]" 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on skipped message = False 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = e77ab909-7680-4e53-a08d-a26ae42f1a8c 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2024-01-16T13:51:12.314 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd mclock profile = high_recovery_ops 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug trim objects = true 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2024-01-16T13:51:12.315 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: bdev async discard = True 2024-01-16T13:51:12.316 INFO:teuthology.orchestra.run.smithi124.stdout: bdev enable discard = True 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore allocator = bitmap 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore block size = 96636764160 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore fsck on mount = True 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluefs = 1/20 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluestore = 1/20 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: debug rocksdb = 4/10 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd full ratio = 0.9 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd nearfull ratio = 0.8 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: osd failsafe full ratio = 0.95 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = bluestore 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.317 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug finisher = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2024-01-16T13:51:12.318 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: # 1m isn't quite enough 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: mon_down_mkfs_grace = 2m 2024-01-16T13:51:12.319 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_filestore_osds = false 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: mon op complaint time = 120 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: client mount timeout = 600 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = "" 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 900 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 900 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2024-01-16T13:51:12.320 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.b] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.c] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.c] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.d] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout:[mds] 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds = 20 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds balancer = 20 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug frag = True 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug scatterstat = True 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: mds op complaint time = 180 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: mds verify scatter = True 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 900 2024-01-16T13:51:12.321 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 900 2024-01-16T13:51:12.323 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T13:51:12.323 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T13:51:12.506 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T13:51:12.506 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:12.506 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T13:51:12.523 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:12.523 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T13:51:12.579 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:51:12.579 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:12.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T13:51:12.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T13:51:12.692 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:12.692 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:12.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:51:12.745 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:12.745 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T13:51:12.745 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T13:51:12.783 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:12.783 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:12.840 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T13:51:12.840 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T13:51:12.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T13:51:12.893 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T13:51:12.896 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T13:51:12.954 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T13:51:12.957 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T13:51:12.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T13:51:13.011 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T13:51:13.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-16T13:51:13.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T13:51:13.133 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T13:51:13.136 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-16T13:51:13.162 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T13:51:13.219 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T13:51:13.222 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-16T13:51:13.249 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T13:51:13.343 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T13:51:13.345 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-16T13:51:13.371 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T13:51:13.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T13:51:13.415 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T13:51:13.428 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-16T13:51:13.511 INFO:teuthology.orchestra.run.smithi113.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-16T13:51:13.524 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-16T13:51:13.568 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-16T13:51:13.581 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-16T13:51:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-16T13:51:13.637 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T13:51:13.637 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T13:51:13.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T13:51:13.663 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:13.663 INFO:tasks.ceph:role: osd.0 2024-01-16T13:51:13.664 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:51:13.664 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T13:51:13.737 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:13.737 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:13.737 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:13.738 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:14.229 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-16T13:51:14.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-16T13:51:14.231 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T13:51:14.365 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T13:51:14.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T13:51:14.469 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:14.469 INFO:tasks.ceph:role: osd.1 2024-01-16T13:51:14.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:51:14.469 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T13:51:14.545 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:14.546 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:15.009 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-16T13:51:15.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-16T13:51:15.011 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T13:51:15.128 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T13:51:15.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T13:51:15.229 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:15.229 INFO:tasks.ceph:role: osd.2 2024-01-16T13:51:15.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:51:15.229 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T13:51:15.305 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:15.305 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:15.305 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:15.305 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:15.306 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:15.306 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:15.306 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:15.306 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:15.306 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:15.306 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:15.787 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-16T13:51:15.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-16T13:51:15.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T13:51:15.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T13:51:15.989 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-16T13:51:16.059 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:16.059 INFO:tasks.ceph:role: osd.3 2024-01-16T13:51:16.059 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:51:16.059 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:16.136 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:16.137 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:16.137 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:16.613 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2024-01-16T13:51:16.614 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2024-01-16T13:51:16.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-16T13:51:16.742 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-16T13:51:16.777 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:16.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:16.884+0000 7f8e70bac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T13:51:16.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:16.889+0000 7f8e70bac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T13:51:16.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:16.889+0000 7f8e70bac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T13:51:16.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:16.889+0000 7f8e70bac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T13:51:16.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:16.889+0000 7f8e70bac640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T13:51:16.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:16.897+0000 7f8e70bac640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T13:51:19.729 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T13:51:19.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:19.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:19.870+0000 7fa8ea5f4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T13:51:19.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:19.870+0000 7fa8ea5f4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T13:51:19.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:19.870+0000 7fa8ea5f4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T13:51:19.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:19.870+0000 7fa8ea5f4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T13:51:19.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:19.870+0000 7fa8ea5f4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T13:51:19.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:19.871+0000 7fa8ea5f4640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T13:51:22.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T13:51:22.732 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:22.841+0000 7fecb11bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T13:51:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:22.841+0000 7fecb11bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T13:51:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:22.841+0000 7fecb11bb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T13:51:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:22.841+0000 7fecb11bb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T13:51:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:22.841+0000 7fecb11bb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T13:51:22.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:22.842+0000 7fecb11bb640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T13:51:25.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T13:51:25.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:25.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:25.811+0000 7fe1b33b7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-16T13:51:25.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:25.811+0000 7fe1b33b7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-16T13:51:25.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:25.811+0000 7fe1b33b7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T13:51:25.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:25.811+0000 7fe1b33b7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T13:51:25.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:25.811+0000 7fe1b33b7640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T13:51:25.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:25.812+0000 7fe1b33b7640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-16T13:51:28.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-16T13:51:28.680 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-16T13:51:28.713 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:28.713 INFO:tasks.ceph:role: osd.4 2024-01-16T13:51:28.713 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:51:28.713 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T13:51:28.792 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:28.792 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:28.792 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:28.793 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:29.229 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2024-01-16T13:51:29.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2024-01-16T13:51:29.231 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-16T13:51:29.413 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-16T13:51:29.448 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-16T13:51:29.516 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:29.516 INFO:tasks.ceph:role: osd.5 2024-01-16T13:51:29.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:51:29.517 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T13:51:29.592 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:29.592 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:29.592 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:29.592 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:29.593 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:29.593 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:29.593 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:29.593 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:29.593 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:29.593 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:30.023 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2024-01-16T13:51:30.025 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2024-01-16T13:51:30.025 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-16T13:51:30.131 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-16T13:51:30.162 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-16T13:51:30.229 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:30.229 INFO:tasks.ceph:role: osd.6 2024-01-16T13:51:30.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:51:30.229 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:30.303 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:30.304 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:30.304 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:30.304 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:30.749 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2024-01-16T13:51:30.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2024-01-16T13:51:30.751 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-16T13:51:30.905 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-16T13:51:30.938 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-16T13:51:31.006 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T13:51:31.006 INFO:tasks.ceph:role: osd.7 2024-01-16T13:51:31.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:51:31.006 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T13:51:31.082 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T13:51:31.083 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T13:51:31.521 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2024-01-16T13:51:31.523 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2024-01-16T13:51:31.523 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-16T13:51:31.701 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-16T13:51:31.739 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:31.838 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:31.854+0000 7fe70234e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-16T13:51:31.838 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:31.855+0000 7fe70234e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-16T13:51:31.838 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:31.855+0000 7fe70234e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T13:51:31.838 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:31.855+0000 7fe70234e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T13:51:31.838 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:31.855+0000 7fe70234e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T13:51:31.839 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:31.855+0000 7fe70234e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-16T13:51:34.696 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-16T13:51:34.729 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:34.824 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:34.841+0000 7fb126c08640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-16T13:51:34.824 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:34.841+0000 7fb126c08640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-16T13:51:34.824 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:34.841+0000 7fb126c08640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T13:51:34.825 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:34.841+0000 7fb126c08640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T13:51:34.825 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:34.841+0000 7fb126c08640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T13:51:34.825 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:34.842+0000 7fb126c08640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-16T13:51:37.679 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-16T13:51:37.712 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:37.810 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:37.827+0000 7f04a16e3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-16T13:51:37.810 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:37.827+0000 7f04a16e3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-16T13:51:37.810 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:37.827+0000 7f04a16e3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T13:51:37.810 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:37.827+0000 7f04a16e3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T13:51:37.811 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:37.827+0000 7f04a16e3640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T13:51:37.811 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:37.828+0000 7f04a16e3640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-16T13:51:40.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:51:40.292 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:51:40.668 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-16T13:51:40.700 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:40.797 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:40.813+0000 7f0e02a20640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-16T13:51:40.797 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:40.814+0000 7f0e02a20640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-16T13:51:40.797 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:40.814+0000 7f0e02a20640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T13:51:40.797 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:40.814+0000 7f0e02a20640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T13:51:40.797 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:40.814+0000 7f0e02a20640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T13:51:40.797 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:40.814+0000 7f0e02a20640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-16T13:51:43.646 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-16T13:51:43.677 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T13:51:43.677 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:43.677 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-16T13:51:43.709 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:43.710 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-16T13:51:43.781 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:43.781 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-16T13:51:43.853 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:43.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T13:51:43.922 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:43.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T13:51:43.991 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:43.991 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T13:51:44.059 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:44.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-16T13:51:44.127 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.127 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T13:51:44.159 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.159 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-16T13:51:44.230 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.230 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-16T13:51:44.300 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.300 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-16T13:51:44.370 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.370 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-16T13:51:44.438 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.438 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-16T13:51:44.505 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.505 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-16T13:51:44.573 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:44.573 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T13:51:44.591 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:44.591 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-16T13:51:44.648 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.649 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-16T13:51:44.666 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:44.666 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-16T13:51:44.722 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T13:51:44.723 DEBUG:teuthology.orchestra.run.smithi113:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T13:51:44.725 DEBUG:teuthology.orchestra.run.smithi124:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T13:51:44.766 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.y] 2024-01-16T13:51:44.766 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDQiaZlJxhjNhAA83zCf3EwB5xwr7O948uvLQ== 2024-01-16T13:51:44.766 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlyDrJARAAz//i2ClH4sbEANRxAiIMlQ== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlwIsPCRAA4q2vkMfBy92/h9eg1gfqRA== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.0] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDUiaZl8s3FNBAA09pce2MNSApF+drjMAkNPA== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.1] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDXiaZlxn/oMxAA5chIuJyqCbNKYh9A1Ta3PQ== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.2] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDaiaZl5hgvMhAAHSCDsDeu8fseIF0mJu4Ctg== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.3] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDdiaZlDZJfMBAAER2AYSt+Qk/a0dwa2/Ij2w== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.x] 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDQiaZlsUoIOhAAlHDQ25EXkUJbqP1AAdaJzw== 2024-01-16T13:51:44.767 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlMdI0DhAAzYRcAH7Jyx48yzJ3Lz3K/g== 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlioyRFRAAlUs/AAsXtP0itYJGyIgSYg== 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.4] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDjiaZlHMALMxAAzgDDZvHvDA+wtSH4Lfrh7Q== 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.5] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDmiaZl4Fk6MhAAlK70uyn3b6DFsjXbBsM0Mw== 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.6] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDpiaZlur1mMRAAzpfPmm4wDxD4Kcex/eBOsA== 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.7] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDsiaZlMtGZMBAAI9uyUo5CajqfhcKH2/xxnQ== 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2024-01-16T13:51:44.768 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlwFPiGRAAc/ztvEiVlF5DA4U3NZB3Og== 2024-01-16T13:51:44.769 INFO:teuthology.orchestra.run.smithi113.stdout:[client.2] 2024-01-16T13:51:44.769 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlNXyXHxAAx9VCUKHDRhkvHMC+VRxGSQ== 2024-01-16T13:51:44.769 INFO:teuthology.orchestra.run.smithi113.stdout:[client.1] 2024-01-16T13:51:44.769 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlxvH/IhAANuKb2btH+PF7v4wMKZaSxw== 2024-01-16T13:51:44.769 INFO:teuthology.orchestra.run.smithi113.stdout:[client.3] 2024-01-16T13:51:44.769 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlT+ZVJhAABoClp/1hNEv4UJ5Drk3JlQ== 2024-01-16T13:51:44.788 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.y] 2024-01-16T13:51:44.788 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDQiaZlJxhjNhAA83zCf3EwB5xwr7O948uvLQ== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlyDrJARAAz//i2ClH4sbEANRxAiIMlQ== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.c] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlwIsPCRAA4q2vkMfBy92/h9eg1gfqRA== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.0] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDUiaZl8s3FNBAA09pce2MNSApF+drjMAkNPA== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.1] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDXiaZlxn/oMxAA5chIuJyqCbNKYh9A1Ta3PQ== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.2] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDaiaZl5hgvMhAAHSCDsDeu8fseIF0mJu4Ctg== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.3] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDdiaZlDZJfMBAAER2AYSt+Qk/a0dwa2/Ij2w== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.x] 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDQiaZlsUoIOhAAlHDQ25EXkUJbqP1AAdaJzw== 2024-01-16T13:51:44.789 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlMdI0DhAAzYRcAH7Jyx48yzJ3Lz3K/g== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.d] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlioyRFRAAlUs/AAsXtP0itYJGyIgSYg== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.4] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDjiaZlHMALMxAAzgDDZvHvDA+wtSH4Lfrh7Q== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.5] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDmiaZl4Fk6MhAAlK70uyn3b6DFsjXbBsM0Mw== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.6] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDpiaZlur1mMRAAzpfPmm4wDxD4Kcex/eBOsA== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.7] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDsiaZlMtGZMBAAI9uyUo5CajqfhcKH2/xxnQ== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlwFPiGRAAc/ztvEiVlF5DA4U3NZB3Og== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[client.2] 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlNXyXHxAAx9VCUKHDRhkvHMC+VRxGSQ== 2024-01-16T13:51:44.790 INFO:teuthology.orchestra.run.smithi124.stdout:[client.1] 2024-01-16T13:51:44.791 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlxvH/IhAANuKb2btH+PF7v4wMKZaSxw== 2024-01-16T13:51:44.791 INFO:teuthology.orchestra.run.smithi124.stdout:[client.3] 2024-01-16T13:51:44.791 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQDRiaZlT+ZVJhAABoClp/1hNEv4UJ5Drk3JlQ== 2024-01-16T13:51:44.791 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T13:51:44.812 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T13:51:44.885 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:44.887 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:44.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:44.946 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:45.002 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.004 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.058 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.060 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.115 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.168 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.170 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.225 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T13:51:45.227 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T13:51:45.280 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:45.282 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:45.339 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:45.341 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T13:51:45.398 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.400 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.456 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.511 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.567 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T13:51:45.620 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.623 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.682 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.741 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.797 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.799 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T13:51:45.855 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T13:51:45.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T13:51:45.882 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T13:51:45.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T13:51:46.007 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-16T13:51:46.034 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T13:51:46.132 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-16T13:51:46.158 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-16T13:51:46.225 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T13:51:46.324 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-16T13:51:46.350 DEBUG:teuthology.orchestra.run.smithi113:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:46.352 DEBUG:teuthology.orchestra.run.smithi124:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T13:51:46.407 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T13:51:46.408 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T13:51:46.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T13:51:46.410 INFO:tasks.ceph.mon.a:Started 2024-01-16T13:51:46.410 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-16T13:51:46.410 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-16T13:51:46.450 INFO:tasks.ceph.mon.b:Started 2024-01-16T13:51:46.450 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-16T13:51:46.450 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-16T13:51:46.452 INFO:tasks.ceph.mon.c:Started 2024-01-16T13:51:46.452 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T13:51:46.452 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-16T13:51:46.452 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-16T13:51:46.455 INFO:tasks.ceph.mgr.y:Started 2024-01-16T13:51:46.455 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T13:51:46.455 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T13:51:46.457 INFO:tasks.ceph.mgr.x:Started 2024-01-16T13:51:46.457 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T13:51:46.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T13:51:46.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.600+0000 7fd2f7186700 1 Processor -- start 2024-01-16T13:51:46.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.600+0000 7fd2f7186700 1 -- start start 2024-01-16T13:51:46.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f7186700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 0x7fd2f006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f7186700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f015e860 con 0x7fd2f006aff0 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f7186700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f00cf490 con 0x7fd2f006c550 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f7186700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f00701c0 con 0x7fd2f015e480 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f4f22700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 0x7fd2f006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f5723700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fd2f006aff0 0x7fd2f006b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:59314/0 (socket says 172.21.15.113:59314) 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f5723700 1 -- 172.21.15.113:0/3521745709 learned_addr learned my addr 172.21.15.113:0/3521745709 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f4f22700 1 -- 172.21.15.113:0/3521745709 >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 msgr2=0x7fd2f006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f4f22700 1 -- 172.21.15.113:0/3521745709 >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 msgr2=0x7fd2f006ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f4f22700 1 --2- 172.21.15.113:0/3521745709 >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 0x7fd2f006ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T13:51:46.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.601+0000 7fd2f4f22700 1 --2- 172.21.15.113:0/3521745709 >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 0x7fd2f006ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T13:51:46.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.778+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638917597 0 0) 0x7fd2f015e860 con 0x7fd2f006aff0 2024-01-16T13:51:46.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.778+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e00035b0 con 0x7fd2f006aff0 2024-01-16T13:51:46.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.778+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363156513 0 0) 0x7fd2f00cf490 con 0x7fd2f006c550 2024-01-16T13:51:46.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.778+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f015e860 con 0x7fd2f006c550 2024-01-16T13:51:46.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.1 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1478364244 0 0) 0x7fd2f015e860 con 0x7fd2f006c550 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 msgr2=0x7fd2f006ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 --2- 172.21.15.113:0/3521745709 >> v2:172.21.15.113:3300/0 conn(0x7fd2f015e480 0x7fd2f006ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 >> v1:172.21.15.113:6789/0 conn(0x7fd2f006aff0 legacy=0x7fd2f006b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f006cb40 con 0x7fd2f006c550 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.1 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fd2dc001670 con 0x7fd2f006c550 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 >> v1:172.21.15.124:6789/0 conn(0x7fd2f006c550 legacy=0x7fd2f006f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2e0003b90 0x7fd2e0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0004850 0x7fd2e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0005330 0x7fd2e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00701c0 con 0x7fd2e0005330 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e000c1e0 con 0x7fd2e0003b90 2024-01-16T13:51:46.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e000c3a0 con 0x7fd2e0004850 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0005330 0x7fd2e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2effff700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0004850 0x7fd2e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f4f22700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2e0003b90 0x7fd2e0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 -- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0004850 msgr2=0x7fd2e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0004850 0x7fd2e0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 -- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2e0003b90 msgr2=0x7fd2e0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2e0003b90 0x7fd2e0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 -- 172.21.15.113:0/3521745709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f015e860 con 0x7fd2e0005330 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2effff700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0004850 0x7fd2e0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 -- 172.21.15.113:0/3521745709 reap_dead start 2024-01-16T13:51:46.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.779+0000 7fd2f5723700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0005330 0x7fd2e000ba10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8002230 tx=0x7fd2e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf38174efa30b9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8001f30 con 0x7fd2e0005330 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e800ce80 con 0x7fd2e0005330 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2ef7fe700 1 -- 172.21.15.113:0/3521745709 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8012640 con 0x7fd2e0005330 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 -- 172.21.15.113:0/3521745709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0005330 msgr2=0x7fd2e000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0005330 0x7fd2e000ba10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8002230 tx=0x7fd2e0005880 comp rx=0 tx=0).stop 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 -- 172.21.15.113:0/3521745709 shutdown_connections 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 --2- 172.21.15.113:0/3521745709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0005330 0x7fd2e000ba10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 -- 172.21.15.113:0/3521745709 >> 172.21.15.113:0/3521745709 conn(0x7fd2f00b37f0 msgr2=0x7fd2f00b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 -- 172.21.15.113:0/3521745709 shutdown_connections 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.781+0000 7fd2f7186700 1 -- 172.21.15.113:0/3521745709 wait complete. 2024-01-16T13:51:46.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.783+0000 7fd2f7186700 1 Processor -- start 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.783+0000 7fd2f7186700 1 -- start start 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f7186700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0005330 0x7fd2f014f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f7186700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f015e9f0 0x7fd2f015ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f7186700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 0x7fd2f00b7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f7186700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00b7860 con 0x7fd2f00b6e70 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f7186700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00c5fd0 con 0x7fd2f015e9f0 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f7186700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f015f4b0 con 0x7fd2e0005330 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 0x7fd2f00b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f4f22700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0005330 0x7fd2f014f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 0x7fd2f00b7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54228/0 (socket says 172.21.15.113:54228) 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 -- 172.21.15.113:0/103585119 learned_addr learned my addr 172.21.15.113:0/103585119 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2effff700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f015e9f0 0x7fd2f015ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 -- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0005330 msgr2=0x7fd2f014f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0005330 0x7fd2f014f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 -- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f015e9f0 msgr2=0x7fd2f015ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:46.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f015e9f0 0x7fd2f015ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 -- 172.21.15.113:0/103585119 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f015f790 con 0x7fd2f00b6e70 2024-01-16T13:51:46.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.784+0000 7fd2f5723700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 0x7fd2f00b7320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0073270 tx=0x7fd2e8001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8012e30 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8001360 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f015fa70 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e801e650 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f00b4bc0 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd2e801e7b0 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.786+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd2e8029640 con 0x7fd2f00b6e70 2024-01-16T13:51:46.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.787+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d00052f0 con 0x7fd2f00b6e70 2024-01-16T13:51:46.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.789+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd2e8017070 con 0x7fd2f00b6e70 2024-01-16T13:51:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.869+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd2d0005870 con 0x7fd2f00b6e70 2024-01-16T13:51:46.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.870+0000 7fd2ed7fa700 1 -- 172.21.15.113:0/103585119 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd2e8002090 con 0x7fd2f00b6e70 2024-01-16T13:51:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 msgr2=0x7fd2f00b7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 0x7fd2f00b7320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0073270 tx=0x7fd2e8001650 comp rx=0 tx=0).stop 2024-01-16T13:51:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 shutdown_connections 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2f00b6e70 0x7fd2f00b7320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd2e0005330 0x7fd2f014f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 --2- 172.21.15.113:0/103585119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f015e9f0 0x7fd2f015ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 >> 172.21.15.113:0/103585119 conn(0x7fd2f00b37f0 msgr2=0x7fd2f00cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 shutdown_connections 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:46.871+0000 7fd2f7186700 1 -- 172.21.15.113:0/103585119 wait complete. 2024-01-16T13:51:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:adjusted tunables profile to default 2024-01-16T13:51:46.866 INFO:tasks.ceph:check_enable_crimson: False 2024-01-16T13:51:46.866 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T13:51:46.866 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:46.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T13:51:46.893 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:46.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T13:51:46.959 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:46.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T13:51:47.027 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:51:47.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-16T13:51:47.093 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:47.093 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-16T13:51:47.121 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:47.121 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-16T13:51:47.182 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.199+0000 7f2693950200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T13:51:47.187 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:47.187 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-16T13:51:47.204 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.221+0000 7fa6132c9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T13:51:47.253 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:51:47.253 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-16T13:51:47.284 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.301+0000 7f2693950200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T13:51:47.294 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.311+0000 7fa6132c9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T13:51:47.309 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 4429388e-5550-4ec2-a13b-35e14274a793 0 2024-01-16T13:51:47.360 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.377+0000 7f2693950200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T13:51:47.367 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.384+0000 7fa6132c9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T13:51:47.434 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.451+0000 7f2693950200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T13:51:47.436 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.454+0000 7fa6132c9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T13:51:47.473 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.490+0000 7f7aeef68700 1 Processor -- start 2024-01-16T13:51:47.473 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.490+0000 7f7aeef68700 1 -- start start 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aeef68700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae806d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aeef68700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ae80722a0 0x7f7ae806b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aeef68700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae806de50 con 0x7f7ae815bcd0 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aeef68700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae80ceb80 con 0x7f7ae80722a0 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aeef68700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80d05b0 con 0x7f7ae81554d0 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ae80722a0 0x7f7ae806b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aecd04700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aecd04700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae806d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57582/0 (socket says 172.21.15.124:57582) 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ae80722a0 0x7f7ae806b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55810/0 (socket says 172.21.15.124:55810) 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aecd04700 1 -- 172.21.15.124:0/4049528354 learned_addr learned my addr 172.21.15.124:0/4049528354 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:47.475 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 -- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 msgr2=0x7f7ae806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 --2- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae806d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 -- 172.21.15.124:0/4049528354 >> v1:172.21.15.113:6789/0 conn(0x7f7ae815bcd0 legacy=0x7f7ae8071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 -- 172.21.15.124:0/4049528354 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae80d0890 con 0x7f7ae80722a0 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aecd04700 1 --2- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7aed505700 1 --2- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae806b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009d90 tx=0x7f7ae000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16d2ffcf11e537e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.491+0000 7f7ae77fe700 1 -- 172.21.15.124:0/4049528354 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae000e070 con 0x7f7ae80722a0 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7ae77fe700 1 -- 172.21.15.124:0/4049528354 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae000bb20 con 0x7f7ae80722a0 2024-01-16T13:51:47.476 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7ae77fe700 1 -- 172.21.15.124:0/4049528354 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0012430 con 0x7f7ae80722a0 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 -- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 msgr2=0x7f7ae806b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 --2- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae806b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009d90 tx=0x7f7ae000b330 comp rx=0 tx=0).stop 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 -- 172.21.15.124:0/4049528354 shutdown_connections 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 --2- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae806b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 --2- 172.21.15.124:0/4049528354 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 -- 172.21.15.124:0/4049528354 >> 172.21.15.124:0/4049528354 conn(0x7f7ae80cd6b0 msgr2=0x7f7ae80cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 -- 172.21.15.124:0/4049528354 shutdown_connections 2024-01-16T13:51:47.479 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.492+0000 7f7aeef68700 1 -- 172.21.15.124:0/4049528354 wait complete. 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 Processor -- start 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 -- start start 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae8156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae80b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae815bcd0 0x7f7ae8148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80b1ca0 con 0x7f7ae815bcd0 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae8156ae0 con 0x7f7ae80722a0 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.494+0000 7f7aeef68700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80b10f0 con 0x7f7ae81554d0 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae8156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7ae7fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae80b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae8156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55814/0 (socket says 172.21.15.124:55814) 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aed505700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae815bcd0 0x7f7ae8148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 -- 172.21.15.124:0/2996333695 learned_addr learned my addr 172.21.15.124:0/2996333695 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 -- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 msgr2=0x7f7ae80b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae80b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.480 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 -- 172.21.15.124:0/2996333695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae815bcd0 msgr2=0x7f7ae8148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae815bcd0 0x7f7ae8148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 -- 172.21.15.124:0/2996333695 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc009550 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7ae7fff700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae80b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aecd04700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae8156290 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00ebe0 tx=0x7f7adc00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc00c850 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc00c9b0 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae0009a70 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc00cd40 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae80b18c0 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.495+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7adc005d20 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.496+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7adc0137b0 con 0x7f7ae80722a0 2024-01-16T13:51:47.481 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.496+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae80b1f10 con 0x7f7ae80722a0 2024-01-16T13:51:47.483 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.500+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7adc021060 con 0x7f7ae80722a0 2024-01-16T13:51:47.568 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.585+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4429388e-5550-4ec2-a13b-35e14274a793", "id": 0} v 0) v1 -- 0x7f7ae80b1410 con 0x7f7ae80722a0 2024-01-16T13:51:47.574 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.591+0000 7f7ae57fa700 1 -- 172.21.15.124:0/2996333695 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4429388e-5550-4ec2-a13b-35e14274a793", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7adc029020 con 0x7f7ae80722a0 2024-01-16T13:51:47.574 INFO:teuthology.orchestra.run.smithi124.stdout:0 2024-01-16T13:51:47.575 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 msgr2=0x7f7ae8156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.575 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae8156290 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00ebe0 tx=0x7f7adc00eef0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.575 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 shutdown_connections 2024-01-16T13:51:47.575 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae815bcd0 0x7f7ae8148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.575 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ae80722a0 0x7f7ae8156290 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.575 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 --2- 172.21.15.124:0/2996333695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ae81554d0 0x7f7ae80b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.576 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 >> 172.21.15.124:0/2996333695 conn(0x7f7ae80cd6b0 msgr2=0x7f7ae80cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:47.576 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 shutdown_connections 2024-01-16T13:51:47.576 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.592+0000 7f7aeef68700 1 -- 172.21.15.124:0/2996333695 wait complete. 2024-01-16T13:51:47.586 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 1f608d38-5fb0-450e-880c-5eaebd9d045a 1 2024-01-16T13:51:47.595 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.612+0000 7fa6132c9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T13:51:47.604 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.620+0000 7f2693950200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T13:51:47.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.742+0000 7f08d6263700 1 Processor -- start 2024-01-16T13:51:47.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.742+0000 7f08d6263700 1 -- start start 2024-01-16T13:51:47.727 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08d6263700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 0x7f08d006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.727 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08d6263700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d006d980 con 0x7f08d006e420 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08d6263700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d00affb0 con 0x7f08d00afb30 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08d6263700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00b0260 con 0x7f08d015e110 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 0x7f08d006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 0x7f08d006f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57600/0 (socket says 172.21.15.124:57600) 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 -- 172.21.15.124:0/580882414 learned_addr learned my addr 172.21.15.124:0/580882414 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cf7fe700 1 --1- 172.21.15.124:0/580882414 >> v1:172.21.15.124:6789/0 conn(0x7f08d00afb30 0x7f08d006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39422/0 (socket says 172.21.15.124:39422) 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08ceffd700 1 -- 172.21.15.124:0/580882414 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233178040 0 0) 0x7f08d00affb0 con 0x7f08d00afb30 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 -- 172.21.15.124:0/580882414 >> v1:172.21.15.124:6789/0 conn(0x7f08d00afb30 legacy=0x7f08d006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 -- 172.21.15.124:0/580882414 >> v1:172.21.15.113:6789/0 conn(0x7f08d006e420 legacy=0x7f08d0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 -- 172.21.15.124:0/580882414 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d00b0540 con 0x7f08d015e110 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08cffff700 1 --2- 172.21.15.124:0/580882414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 0x7f08d006f680 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009920 tx=0x7f08c0009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52cdf409d36b046d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08ceffd700 1 -- 172.21.15.124:0/580882414 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0004230 con 0x7f08d015e110 2024-01-16T13:51:47.728 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.743+0000 7f08ceffd700 1 -- 172.21.15.124:0/580882414 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c0004390 con 0x7f08d015e110 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08ceffd700 1 -- 172.21.15.124:0/580882414 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0009e70 con 0x7f08d015e110 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 -- 172.21.15.124:0/580882414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 msgr2=0x7f08d006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 --2- 172.21.15.124:0/580882414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 0x7f08d006f680 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009920 tx=0x7f08c0009c30 comp rx=0 tx=0).stop 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 -- 172.21.15.124:0/580882414 shutdown_connections 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 --2- 172.21.15.124:0/580882414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d015e110 0x7f08d006f680 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 -- 172.21.15.124:0/580882414 >> 172.21.15.124:0/580882414 conn(0x7f08d00b3250 msgr2=0x7f08d00d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 -- 172.21.15.124:0/580882414 shutdown_connections 2024-01-16T13:51:47.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.744+0000 7f08d6263700 1 -- 172.21.15.124:0/580882414 wait complete. 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.745+0000 7f08d6263700 1 Processor -- start 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 -- start start 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 0x7f08d00cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 0x7f08d00c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08d015e110 0x7f08d00b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00d0470 con 0x7f08d015e110 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01f6fb0 con 0x7f08d00afb30 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08d6263700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01f7290 con 0x7f08d006e420 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08cf7fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 0x7f08d00c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08cffff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 0x7f08d00cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08cf7fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 0x7f08d00c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55820/0 (socket says 172.21.15.124:55820) 2024-01-16T13:51:47.732 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08cffff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 0x7f08d00cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57612/0 (socket says 172.21.15.124:57612) 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.746+0000 7f08cf7fe700 1 -- 172.21.15.124:0/1787336389 learned_addr learned my addr 172.21.15.124:0/1787336389 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08d4800700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08d015e110 0x7f08d00b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cffff700 1 -- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 msgr2=0x7f08d00c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cffff700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 0x7f08d00c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cffff700 1 -- 172.21.15.124:0/1787336389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08d015e110 msgr2=0x7f08d00b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cffff700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08d015e110 0x7f08d00b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cffff700 1 -- 172.21.15.124:0/1787336389 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0009550 con 0x7f08d006e420 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cf7fe700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 0x7f08d00c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08cffff700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 0x7f08d00cff30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f08c0015000 tx=0x7f08c0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c000be30 con 0x7f08d006e420 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c00036a0 con 0x7f08d006e420 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0003a30 con 0x7f08d006e420 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d006b880 con 0x7f08d006e420 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d01f7850 con 0x7f08d006e420 2024-01-16T13:51:47.733 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f08c0021070 con 0x7f08d006e420 2024-01-16T13:51:47.734 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.747+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f08c001a440 con 0x7f08d006e420 2024-01-16T13:51:47.734 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.748+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d0069020 con 0x7f08d006e420 2024-01-16T13:51:47.735 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.752+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f08c0029020 con 0x7f08d006e420 2024-01-16T13:51:47.747 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.764+0000 7fa6132c9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T13:51:47.768 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.785+0000 7f2693950200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T13:51:47.822 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.839+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1f608d38-5fb0-450e-880c-5eaebd9d045a", "id": 1} v 0) v1 -- 0x7f08d0069020 con 0x7f08d006e420 2024-01-16T13:51:47.828 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.845+0000 7f08ccff9700 1 -- 172.21.15.124:0/1787336389 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f608d38-5fb0-450e-880c-5eaebd9d045a", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f08c0024470 con 0x7f08d006e420 2024-01-16T13:51:47.828 INFO:teuthology.orchestra.run.smithi124.stdout:1 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 msgr2=0x7f08d00cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 0x7f08d00cff30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f08c0015000 tx=0x7f08c0004750 comp rx=0 tx=0).stop 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 shutdown_connections 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08d015e110 0x7f08d00b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f08d006e420 0x7f08d00cff30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 --2- 172.21.15.124:0/1787336389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08d00afb30 0x7f08d00c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 >> 172.21.15.124:0/1787336389 conn(0x7f08d00b3250 msgr2=0x7f08d00b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 shutdown_connections 2024-01-16T13:51:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.846+0000 7f08d6263700 1 -- 172.21.15.124:0/1787336389 wait complete. 2024-01-16T13:51:47.842 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 90bfcd09-fa8a-4659-a24e-fbfb5f73cd90 2 2024-01-16T13:51:47.858 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.875+0000 7fa6132c9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T13:51:47.884 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:47.901+0000 7f2693950200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T13:51:47.952 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:47.969+0000 7fa6132c9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T13:51:47.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.980+0000 7f9cf4b64700 1 Processor -- start 2024-01-16T13:51:47.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.980+0000 7f9cf4b64700 1 -- start start 2024-01-16T13:51:47.965 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cf4b64700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9cf00cdbc0 0x7f9cf00d0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cf4b64700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9cf006fe50 0x7f9cf00ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cf4b64700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf006edd0 con 0x7f9cf00d0d90 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cf4b64700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf006f090 con 0x7f9cf00cdbc0 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cf4b64700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf006f370 con 0x7f9cf006fe50 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cedd9b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9cf006fe50 0x7f9cf00ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9ceed9d700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f9cf00d0d90 0x7f9cf015d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39436/0 (socket says 172.21.15.124:39436) 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9ceed9d700 1 -- 172.21.15.124:0/3060437424 learned_addr learned my addr 172.21.15.124:0/3060437424 (peer_addr_for_me v1:172.21.15.124:0/0) 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cedd9b700 1 --2- 172.21.15.124:0/3060437424 >> v2:172.21.15.124:3300/0 conn(0x7f9cf006fe50 0x7f9cf00ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55822/0 (socket says 172.21.15.124:55822) 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cee59c700 1 --2- 172.21.15.124:0/3060437424 >> v2:172.21.15.113:3300/0 conn(0x7f9cf00cdbc0 0x7f9cf00d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cedd9b700 1 -- 172.21.15.124:0/3060437424 >> v2:172.21.15.113:3300/0 conn(0x7f9cf00cdbc0 msgr2=0x7f9cf00d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cedd9b700 1 --2- 172.21.15.124:0/3060437424 >> v2:172.21.15.113:3300/0 conn(0x7f9cf00cdbc0 0x7f9cf00d0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cedd9b700 1 -- 172.21.15.124:0/3060437424 >> v1:172.21.15.124:6789/0 conn(0x7f9cf00d0d90 legacy=0x7f9cf015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9ced59a700 1 -- 172.21.15.124:0/3060437424 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283768109 0 0) 0x7f9cf006edd0 con 0x7f9cf00d0d90 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.981+0000 7f9cedd9b700 1 -- 172.21.15.124:0/3060437424 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf006f650 con 0x7f9cf006fe50 2024-01-16T13:51:47.966 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cedd9b700 1 --2- 172.21.15.124:0/3060437424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf00ce450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0071a80 tx=0x7f9ce4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c6aff0b17710b0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9ced59a700 1 -- 172.21.15.124:0/3060437424 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4011830 con 0x7f9cf006fe50 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9ced59a700 1 -- 172.21.15.124:0/3060437424 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce4011990 con 0x7f9cf006fe50 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9ced59a700 1 -- 172.21.15.124:0/3060437424 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4011d20 con 0x7f9cf006fe50 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3060437424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 msgr2=0x7f9cf00ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3060437424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf00ce450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0071a80 tx=0x7f9ce4009b20 comp rx=0 tx=0).stop 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3060437424 shutdown_connections 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3060437424 >> v2:172.21.15.113:3300/0 conn(0x7f9cf00cdbc0 0x7f9cf00d0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3060437424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf00ce450 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3060437424 >> 172.21.15.124:0/3060437424 conn(0x7f9cf00b3780 msgr2=0x7f9cf00b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3060437424 shutdown_connections 2024-01-16T13:51:47.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.982+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3060437424 wait complete. 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 Processor -- start 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 -- start start 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cf00cdbc0 0x7f9cf00b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9cf00d0d90 0x7f9cf015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf00b2fd0 con 0x7f9cf00cdbc0 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf014c5e0 con 0x7f9cf006fe50 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.984+0000 7f9cf4b64700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf00b2520 con 0x7f9cf00d0d90 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf014c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55830/0 (socket says 172.21.15.124:55830) 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 -- 172.21.15.124:0/3237544140 learned_addr learned my addr 172.21.15.124:0/3237544140 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9ceed9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9cf00d0d90 0x7f9cf015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cedd9b700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cf00cdbc0 0x7f9cf00b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 -- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9cf00d0d90 msgr2=0x7f9cf015c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.970 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9cf00d0d90 0x7f9cf015c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 -- 172.21.15.124:0/3237544140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cf00cdbc0 msgr2=0x7f9cf00b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cf00cdbc0 0x7f9cf00b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 -- 172.21.15.124:0/3237544140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40094a0 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cee59c700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf014c0a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000e9b0 tx=0x7f9ce000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0009710 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce000ee50 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf015d440 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000ca50 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf00b2a10 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9ce0005a20 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.985+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f9ce00067c0 con 0x7f9cf006fe50 2024-01-16T13:51:47.971 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.986+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf0069020 con 0x7f9cf006fe50 2024-01-16T13:51:47.972 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:47.988+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9ce0021060 con 0x7f9cf006fe50 2024-01-16T13:51:47.984 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:48.002+0000 7f2693950200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T13:51:48.028 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:48.046+0000 7fa6132c9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T13:51:48.055 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.072+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "90bfcd09-fa8a-4659-a24e-fbfb5f73cd90", "id": 2} v 0) v1 -- 0x7f9cf00b2db0 con 0x7f9cf006fe50 2024-01-16T13:51:48.061 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.078+0000 7f9cdeffd700 1 -- 172.21.15.124:0/3237544140 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90bfcd09-fa8a-4659-a24e-fbfb5f73cd90", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9ce0017070 con 0x7f9cf006fe50 2024-01-16T13:51:48.061 INFO:teuthology.orchestra.run.smithi124.stdout:2 2024-01-16T13:51:48.062 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 msgr2=0x7f9cf014c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.062 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf014c0a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000e9b0 tx=0x7f9ce000ecc0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.062 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 shutdown_connections 2024-01-16T13:51:48.062 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9cf00d0d90 0x7f9cf015c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.062 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf006fe50 0x7f9cf014c0a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.062 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 --2- 172.21.15.124:0/3237544140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cf00cdbc0 0x7f9cf00b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 >> 172.21.15.124:0/3237544140 conn(0x7f9cf00b3780 msgr2=0x7f9cf00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 shutdown_connections 2024-01-16T13:51:48.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.079+0000 7f9cf4b64700 1 -- 172.21.15.124:0/3237544140 wait complete. 2024-01-16T13:51:48.063 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:48.080+0000 7f2693950200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T13:51:48.074 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 391ffd90-1e6a-4e72-adc3-6ee6320a0913 3 2024-01-16T13:51:48.247 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:48.265+0000 7fa6132c9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T13:51:48.260 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.277+0000 7f19a0f2e700 1 Processor -- start 2024-01-16T13:51:48.260 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.277+0000 7f19a0f2e700 1 -- start start 2024-01-16T13:51:48.262 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f19a0f2e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 0x7f199c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.262 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f19a0f2e700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199c162af0 con 0x7f199c0d3710 2024-01-16T13:51:48.262 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f19a0f2e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199c0d13e0 con 0x7f199c0b98e0 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f19a0f2e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0b67f0 con 0x7f199c161510 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199a59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 0x7f199c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f1999d9b700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f199c0b98e0 0x7f199c0b59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39444/0 (socket says 172.21.15.124:39444) 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199a59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 0x7f199c0cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57642/0 (socket says 172.21.15.124:57642) 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f1999d9b700 1 -- 172.21.15.124:0/1734335676 learned_addr learned my addr 172.21.15.124:0/1734335676 (peer_addr_for_me v1:172.21.15.124:0/0) 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199a59c700 1 -- 172.21.15.124:0/1734335676 >> v1:172.21.15.124:6789/0 conn(0x7f199c0b98e0 legacy=0x7f199c0b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199a59c700 1 -- 172.21.15.124:0/1734335676 >> v1:172.21.15.113:6789/0 conn(0x7f199c0d3710 legacy=0x7f199c0b60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199a59c700 1 -- 172.21.15.124:0/1734335676 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c161f70 con 0x7f199c161510 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199959a700 1 -- 172.21.15.124:0/1734335676 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796557128 0 0) 0x7f199c0d13e0 con 0x7f199c0b98e0 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199a59c700 1 --2- 172.21.15.124:0/1734335676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 0x7f199c0cec20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f198c009870 tx=0x7f198c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36726bec4a4df303 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199959a700 1 -- 172.21.15.124:0/1734335676 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c011830 con 0x7f199c161510 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199959a700 1 -- 172.21.15.124:0/1734335676 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c011990 con 0x7f199c161510 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.278+0000 7f199959a700 1 -- 172.21.15.124:0/1734335676 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c011d20 con 0x7f199c161510 2024-01-16T13:51:48.263 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 -- 172.21.15.124:0/1734335676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 msgr2=0x7f199c0cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.264 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 --2- 172.21.15.124:0/1734335676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 0x7f199c0cec20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f198c009870 tx=0x7f198c009b80 comp rx=0 tx=0).stop 2024-01-16T13:51:48.264 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 -- 172.21.15.124:0/1734335676 shutdown_connections 2024-01-16T13:51:48.264 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 --2- 172.21.15.124:0/1734335676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c161510 0x7f199c0cec20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.264 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 -- 172.21.15.124:0/1734335676 >> 172.21.15.124:0/1734335676 conn(0x7f199c06b410 msgr2=0x7f199c06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.264 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 -- 172.21.15.124:0/1734335676 shutdown_connections 2024-01-16T13:51:48.264 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.279+0000 7f19a0f2e700 1 -- 172.21.15.124:0/1734335676 wait complete. 2024-01-16T13:51:48.266 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.280+0000 7f19a0f2e700 1 Processor -- start 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.280+0000 7f19a0f2e700 1 -- start start 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f19a0f2e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f199c0b98e0 0x7f199c1533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f19a0f2e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 0x7f199c0cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f19a0f2e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f199c161510 0x7f199c0cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f19a0f2e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c153900 con 0x7f199c0b98e0 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f19a0f2e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c1f3be0 con 0x7f199c161510 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f19a0f2e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199c0c9200 con 0x7f199c0d3710 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f199ad9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f199c161510 0x7f199c0cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f1999d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 0x7f199c0cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f1999d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 0x7f199c0cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57656/0 (socket says 172.21.15.124:57656) 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f199ad9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f199c161510 0x7f199c0cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55842/0 (socket says 172.21.15.124:55842) 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f199a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f199c0b98e0 0x7f199c1533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.267 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.281+0000 7f1999d9b700 1 -- 172.21.15.124:0/3751698134 learned_addr learned my addr 172.21.15.124:0/3751698134 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f1999d9b700 1 -- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f199c161510 msgr2=0x7f199c0cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f1999d9b700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f199c161510 0x7f199c0cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f1999d9b700 1 -- 172.21.15.124:0/3751698134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f199c0b98e0 msgr2=0x7f199c1533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f1999d9b700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f199c0b98e0 0x7f199c1533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f1999d9b700 1 -- 172.21.15.124:0/3751698134 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094a0 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f199a59c700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f199c0b98e0 0x7f199c1533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f1999d9b700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 0x7f199c0cc0d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f199000c520 tx=0x7f199000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990004c80 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1990004e00 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199c0c94e0 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990007310 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199c0c9aa0 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f199001d3f0 con 0x7f199c0d3710 2024-01-16T13:51:48.268 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.282+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f1990013850 con 0x7f199c0d3710 2024-01-16T13:51:48.269 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.283+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c068450 con 0x7f199c0d3710 2024-01-16T13:51:48.269 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.285+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f19900133c0 con 0x7f199c0d3710 2024-01-16T13:51:48.295 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:48.312+0000 7f2693950200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T13:51:48.319 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:48.337+0000 7fa6132c9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T13:51:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.370+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "391ffd90-1e6a-4e72-adc3-6ee6320a0913", "id": 3} v 0) v1 -- 0x7f199c069020 con 0x7f199c0d3710 2024-01-16T13:51:48.358 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.375+0000 7f198affd700 1 -- 172.21.15.124:0/3751698134 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "391ffd90-1e6a-4e72-adc3-6ee6320a0913", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1990013a60 con 0x7f199c0d3710 2024-01-16T13:51:48.358 INFO:teuthology.orchestra.run.smithi124.stdout:3 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 msgr2=0x7f199c0cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 0x7f199c0cc0d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f199000c520 tx=0x7f199000c830 comp rx=0 tx=0).stop 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 shutdown_connections 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f199c161510 0x7f199c0cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f199c0b98e0 0x7f199c1533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 --2- 172.21.15.124:0/3751698134 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f199c0d3710 0x7f199c0cc0d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 >> 172.21.15.124:0/3751698134 conn(0x7f199c06b410 msgr2=0x7f199c160330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 shutdown_connections 2024-01-16T13:51:48.360 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.376+0000 7f19a0f2e700 1 -- 172.21.15.124:0/3751698134 wait complete. 2024-01-16T13:51:48.369 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:48.386+0000 7f2693950200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T13:51:48.371 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 4a6407f5-da2f-4594-8157-eb029642ee09 4 2024-01-16T13:51:48.493 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.510+0000 7f3a2e6cb700 1 Processor -- start 2024-01-16T13:51:48.493 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.510+0000 7f3a2e6cb700 1 -- start start 2024-01-16T13:51:48.495 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a2e6cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280d0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a2e6cb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3a2806fe50 0x7f3a280ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a2e6cb700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2806edd0 con 0x7f3a280d0d90 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a2e6cb700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a2806f090 con 0x7f3a2806fe50 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a2e6cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2806f370 con 0x7f3a280cdbc0 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3a2806fe50 0x7f3a280ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3a2806fe50 0x7f3a280ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55852/0 (socket says 172.21.15.124:55852) 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 -- 172.21.15.124:0/4094748829 learned_addr learned my addr 172.21.15.124:0/4094748829 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a27fff700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 -- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 msgr2=0x7f3a280d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280d0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 -- 172.21.15.124:0/4094748829 >> v1:172.21.15.113:6789/0 conn(0x7f3a280d0d90 legacy=0x7f3a2815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 -- 172.21.15.124:0/4094748829 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2806f650 con 0x7f3a2806fe50 2024-01-16T13:51:48.496 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a27fff700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280d0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a277fe700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a2806fe50 0x7f3a280ce450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0098f0 tx=0x7f3a1c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12badd3d893ac124 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a26ffd700 1 -- 172.21.15.124:0/4094748829 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c011830 con 0x7f3a2806fe50 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a26ffd700 1 -- 172.21.15.124:0/4094748829 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c011990 con 0x7f3a2806fe50 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.511+0000 7f3a26ffd700 1 -- 172.21.15.124:0/4094748829 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c011d20 con 0x7f3a2806fe50 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a2806fe50 msgr2=0x7f3a280ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a2806fe50 0x7f3a280ce450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0098f0 tx=0x7f3a1c009c00 comp rx=0 tx=0).stop 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4094748829 shutdown_connections 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280d0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4094748829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a2806fe50 0x7f3a280ce450 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.497 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4094748829 >> 172.21.15.124:0/4094748829 conn(0x7f3a280b3780 msgr2=0x7f3a280b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.498 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4094748829 shutdown_connections 2024-01-16T13:51:48.498 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.512+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4094748829 wait complete. 2024-01-16T13:51:48.500 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.513+0000 7f3a2e6cb700 1 Processor -- start 2024-01-16T13:51:48.500 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.513+0000 7f3a2e6cb700 1 -- start start 2024-01-16T13:51:48.500 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2e6cb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a2806fe50 0x7f3a2814e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.500 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2e6cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2e6cb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a280d0d90 0x7f3a280b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2e6cb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a280c8a80 con 0x7f3a2806fe50 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2e6cb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2814f9a0 con 0x7f3a280d0d90 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2e6cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2814f000 con 0x7f3a280cdbc0 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a277fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a2cc68700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a280d0d90 0x7f3a280b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a277fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57678/0 (socket says 172.21.15.124:57678) 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a277fe700 1 -- 172.21.15.124:0/4198422362 learned_addr learned my addr 172.21.15.124:0/4198422362 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.514+0000 7f3a27fff700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a2806fe50 0x7f3a2814e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2cc68700 1 -- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 msgr2=0x7f3a280b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2cc68700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2cc68700 1 -- 172.21.15.124:0/4198422362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a2806fe50 msgr2=0x7f3a2814e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2cc68700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a2806fe50 0x7f3a2814e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2cc68700 1 -- 172.21.15.124:0/4198422362 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a180094a0 con 0x7f3a280d0d90 2024-01-16T13:51:48.501 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2cc68700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a280d0d90 0x7f3a280b5da0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a280d1220 tx=0x7f3a2000cdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20003c00 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a20003d60 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c009550 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2000aa20 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2814f5c0 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3a200050d0 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.515+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f3a20005890 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.516+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2814fae0 con 0x7f3a280d0d90 2024-01-16T13:51:48.502 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.518+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3a2814fae0 con 0x7f3a280d0d90 2024-01-16T13:51:48.583 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.600+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a6407f5-da2f-4594-8157-eb029642ee09", "id": 4} v 0) v1 -- 0x7f3a28069020 con 0x7f3a280d0d90 2024-01-16T13:51:48.587 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.604+0000 7f3a24ff9700 1 -- 172.21.15.124:0/4198422362 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a6407f5-da2f-4594-8157-eb029642ee09", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3a20005450 con 0x7f3a280d0d90 2024-01-16T13:51:48.587 INFO:teuthology.orchestra.run.smithi124.stdout:4 2024-01-16T13:51:48.588 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a280d0d90 msgr2=0x7f3a280b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.588 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a280d0d90 0x7f3a280b5da0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3a280d1220 tx=0x7f3a2000cdc0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.588 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 shutdown_connections 2024-01-16T13:51:48.588 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a280d0d90 0x7f3a280b5da0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.588 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a2806fe50 0x7f3a2814e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.588 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 --2- 172.21.15.124:0/4198422362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3a280cdbc0 0x7f3a280b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.589 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 >> 172.21.15.124:0/4198422362 conn(0x7f3a280b3780 msgr2=0x7f3a28061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.589 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 shutdown_connections 2024-01-16T13:51:48.589 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.605+0000 7f3a2e6cb700 1 -- 172.21.15.124:0/4198422362 wait complete. 2024-01-16T13:51:48.599 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new b802e6e8-2700-4a63-a29e-833fe899f1ef 5 2024-01-16T13:51:48.694 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:48.711+0000 7fa6132c9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T13:51:48.722 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.739+0000 7f8534132700 1 Processor -- start 2024-01-16T13:51:48.722 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.739+0000 7f8534132700 1 -- start start 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8534132700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f852c15b9d0 0x7f852c0b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8534132700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c0b4520 0x7f852c071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8534132700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852c0b4a80 con 0x7f852c071960 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8534132700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c071df0 con 0x7f852c15b9d0 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8534132700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0b4d60 con 0x7f852c0b4520 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85316cd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c0b4520 0x7f852c071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85326cf700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f852c071960 0x7f852c161ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39446/0 (socket says 172.21.15.124:39446) 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85326cf700 1 -- 172.21.15.124:0/4093405817 learned_addr learned my addr 172.21.15.124:0/4093405817 (peer_addr_for_me v1:172.21.15.124:0/0) 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8531ece700 1 --2- 172.21.15.124:0/4093405817 >> v2:172.21.15.113:3300/0 conn(0x7f852c15b9d0 0x7f852c0b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8530ecc700 1 -- 172.21.15.124:0/4093405817 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167057343 0 0) 0x7f852c0b4a80 con 0x7f852c071960 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f8530ecc700 1 -- 172.21.15.124:0/4093405817 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85180035b0 con 0x7f852c071960 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85316cd700 1 -- 172.21.15.124:0/4093405817 >> v2:172.21.15.113:3300/0 conn(0x7f852c15b9d0 msgr2=0x7f852c0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85316cd700 1 --2- 172.21.15.124:0/4093405817 >> v2:172.21.15.113:3300/0 conn(0x7f852c15b9d0 0x7f852c0b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85316cd700 1 -- 172.21.15.124:0/4093405817 >> v1:172.21.15.124:6789/0 conn(0x7f852c071960 legacy=0x7f852c161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.740+0000 7f85316cd700 1 -- 172.21.15.124:0/4093405817 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0ae490 con 0x7f852c0b4520 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f85316cd700 1 --2- 172.21.15.124:0/4093405817 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c0b4520 0x7f852c071420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f851c009840 tx=0x7f851c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d46254d2b5d7e856 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8530ecc700 1 -- 172.21.15.124:0/4093405817 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c011830 con 0x7f852c0b4520 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8530ecc700 1 -- 172.21.15.124:0/4093405817 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c011990 con 0x7f852c0b4520 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8530ecc700 1 -- 172.21.15.124:0/4093405817 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c011d20 con 0x7f852c0b4520 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 -- 172.21.15.124:0/4093405817 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c0b4520 msgr2=0x7f852c071420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 --2- 172.21.15.124:0/4093405817 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c0b4520 0x7f852c071420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f851c009840 tx=0x7f851c009b50 comp rx=0 tx=0).stop 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 -- 172.21.15.124:0/4093405817 shutdown_connections 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 --2- 172.21.15.124:0/4093405817 >> v2:172.21.15.113:3300/0 conn(0x7f852c15b9d0 0x7f852c0b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 --2- 172.21.15.124:0/4093405817 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c0b4520 0x7f852c071420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 -- 172.21.15.124:0/4093405817 >> 172.21.15.124:0/4093405817 conn(0x7f852c074170 msgr2=0x7f852c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 -- 172.21.15.124:0/4093405817 shutdown_connections 2024-01-16T13:51:48.726 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.741+0000 7f8534132700 1 -- 172.21.15.124:0/4093405817 wait complete. 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 Processor -- start 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 -- start start 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 0x7f852c0b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852c0b4520 0x7f852c14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f852c15b9d0 0x7f852c14aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c148500 con 0x7f852c0b4520 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0b2040 con 0x7f852c15b9d0 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8534132700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0b2320 con 0x7f852c071960 2024-01-16T13:51:48.729 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8531ece700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 0x7f852c0b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8531ece700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 0x7f852c0b1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57688/0 (socket says 172.21.15.124:57688) 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f85326cf700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f852c15b9d0 0x7f852c14aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f8531ece700 1 -- 172.21.15.124:0/2589259088 learned_addr learned my addr 172.21.15.124:0/2589259088 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.743+0000 7f85316cd700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852c0b4520 0x7f852c14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8531ece700 1 -- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f852c15b9d0 msgr2=0x7f852c14aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8531ece700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f852c15b9d0 0x7f852c14aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8531ece700 1 -- 172.21.15.124:0/2589259088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852c0b4520 msgr2=0x7f852c14a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8531ece700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852c0b4520 0x7f852c14a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8531ece700 1 -- 172.21.15.124:0/2589259088 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c0094a0 con 0x7f852c071960 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8531ece700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 0x7f852c0b1740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f852800e870 tx=0x7f852800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528009590 con 0x7f852c071960 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8528013680 con 0x7f852c071960 2024-01-16T13:51:48.730 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280139f0 con 0x7f852c071960 2024-01-16T13:51:48.731 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c0b2600 con 0x7f852c071960 2024-01-16T13:51:48.731 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.744+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c1f13a0 con 0x7f852c071960 2024-01-16T13:51:48.731 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.745+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8528005a20 con 0x7f852c071960 2024-01-16T13:51:48.731 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.745+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f85280066b0 con 0x7f852c071960 2024-01-16T13:51:48.731 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.745+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c0b27f0 con 0x7f852c071960 2024-01-16T13:51:48.731 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.747+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f8528021060 con 0x7f852c071960 2024-01-16T13:51:48.765 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:48.782+0000 7f2693950200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T13:51:48.816 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.834+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b802e6e8-2700-4a63-a29e-833fe899f1ef", "id": 5} v 0) v1 -- 0x7f852c069020 con 0x7f852c071960 2024-01-16T13:51:48.824 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.841+0000 7f85227fc700 1 -- 172.21.15.124:0/2589259088 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b802e6e8-2700-4a63-a29e-833fe899f1ef", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8528017070 con 0x7f852c071960 2024-01-16T13:51:48.825 INFO:teuthology.orchestra.run.smithi124.stdout:5 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 msgr2=0x7f852c0b1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 0x7f852c0b1740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f852800e870 tx=0x7f852800eb80 comp rx=0 tx=0).stop 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 shutdown_connections 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f852c15b9d0 0x7f852c14aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f852c071960 0x7f852c0b1740 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 --2- 172.21.15.124:0/2589259088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f852c0b4520 0x7f852c14a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 >> 172.21.15.124:0/2589259088 conn(0x7f852c074170 msgr2=0x7f852c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 shutdown_connections 2024-01-16T13:51:48.826 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.842+0000 7f8534132700 1 -- 172.21.15.124:0/2589259088 wait complete. 2024-01-16T13:51:48.837 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new a074ea89-c631-40d6-bb40-703de5636e78 6 2024-01-16T13:51:48.912 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:48.929+0000 7fa6132c9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T13:51:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.976+0000 7fa282b2e700 1 Processor -- start 2024-01-16T13:51:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa282b2e700 1 -- start start 2024-01-16T13:51:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa282b2e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa282b2e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa27c06b700 0x7fa27c0cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa282b2e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c0cec10 con 0x7fa27c0af860 2024-01-16T13:51:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa282b2e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c15c0f0 con 0x7fa27c06b700 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa282b2e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c071e90 con 0x7fa27c15bd10 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa2808ca700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.977+0000 7fa27bfff700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa27c0af860 0x7fa27c06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39462/0 (socket says 172.21.15.124:39462) 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa27bfff700 1 -- 172.21.15.124:0/2415250173 learned_addr learned my addr 172.21.15.124:0/2415250173 (peer_addr_for_me v1:172.21.15.124:0/0) 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa2810cb700 1 --2- 172.21.15.124:0/2415250173 >> v2:172.21.15.113:3300/0 conn(0x7fa27c06b700 0x7fa27c0cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa2808ca700 1 -- 172.21.15.124:0/2415250173 >> v2:172.21.15.113:3300/0 conn(0x7fa27c06b700 msgr2=0x7fa27c0cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa2808ca700 1 --2- 172.21.15.124:0/2415250173 >> v2:172.21.15.113:3300/0 conn(0x7fa27c06b700 0x7fa27c0cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa2808ca700 1 -- 172.21.15.124:0/2415250173 >> v1:172.21.15.124:6789/0 conn(0x7fa27c0af860 legacy=0x7fa27c06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa27b7fe700 1 -- 172.21.15.124:0/2415250173 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260801501 0 0) 0x7fa27c0cec10 con 0x7fa27c0af860 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa2808ca700 1 -- 172.21.15.124:0/2415250173 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c072150 con 0x7fa27c15bd10 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa2808ca700 1 --2- 172.21.15.124:0/2415250173 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c06d440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa270009840 tx=0x7fa270009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bdc72ae1c6932a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa27b7fe700 1 -- 172.21.15.124:0/2415250173 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270011830 con 0x7fa27c15bd10 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa27b7fe700 1 -- 172.21.15.124:0/2415250173 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa270011990 con 0x7fa27c15bd10 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.978+0000 7fa27b7fe700 1 -- 172.21.15.124:0/2415250173 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270011d20 con 0x7fa27c15bd10 2024-01-16T13:51:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 -- 172.21.15.124:0/2415250173 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 msgr2=0x7fa27c06d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 --2- 172.21.15.124:0/2415250173 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c06d440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa270009840 tx=0x7fa270009b50 comp rx=0 tx=0).stop 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 -- 172.21.15.124:0/2415250173 shutdown_connections 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 --2- 172.21.15.124:0/2415250173 >> v2:172.21.15.113:3300/0 conn(0x7fa27c06b700 0x7fa27c0cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 --2- 172.21.15.124:0/2415250173 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c06d440 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 -- 172.21.15.124:0/2415250173 >> 172.21.15.124:0/2415250173 conn(0x7fa27c0b20c0 msgr2=0x7fa27c0b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 -- 172.21.15.124:0/2415250173 shutdown_connections 2024-01-16T13:51:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.979+0000 7fa282b2e700 1 -- 172.21.15.124:0/2415250173 wait complete. 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.980+0000 7fa282b2e700 1 Processor -- start 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.980+0000 7fa282b2e700 1 -- start start 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa282b2e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 0x7fa27c0ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa282b2e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa27c0af860 0x7fa27c151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa282b2e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa282b2e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c158a30 con 0x7fa27c0af860 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa282b2e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0aa830 con 0x7fa27c06b700 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa282b2e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c158c80 con 0x7fa27c15bd10 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 0x7fa27c0ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2810cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 0x7fa27c0ab810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55882/0 (socket says 172.21.15.124:55882) 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 -- 172.21.15.124:0/1374527184 learned_addr learned my addr 172.21.15.124:0/1374527184 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2810cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:57710/0 (socket says 172.21.15.124:57710) 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa27bfff700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa27c0af860 0x7fa27c151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 -- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 msgr2=0x7fa27c151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 -- 172.21.15.124:0/1374527184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa27c0af860 msgr2=0x7fa27c151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa27c0af860 0x7fa27c151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 -- 172.21.15.124:0/1374527184 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2700094a0 con 0x7fa27c06b700 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2810cb700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.981+0000 7fa2808ca700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 0x7fa27c0ab810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa270009fd0 tx=0x7fa270004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2700036a0 con 0x7fa27c06b700 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27001f070 con 0x7fa27c06b700 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c158f60 con 0x7fa27c06b700 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270003a60 con 0x7fa27c06b700 2024-01-16T13:51:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c0abd50 con 0x7fa27c06b700 2024-01-16T13:51:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa270026480 con 0x7fa27c06b700 2024-01-16T13:51:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.982+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fa270021070 con 0x7fa27c06b700 2024-01-16T13:51:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.983+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c159480 con 0x7fa27c06b700 2024-01-16T13:51:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:48.985+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa27c159480 con 0x7fa27c06b700 2024-01-16T13:51:48.997 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:49.015+0000 7f2693950200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T13:51:49.053 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.070+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a074ea89-c631-40d6-bb40-703de5636e78", "id": 6} v 0) v1 -- 0x7fa27c15b530 con 0x7fa27c06b700 2024-01-16T13:51:49.061 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.078+0000 7fa2797fa700 1 -- 172.21.15.124:0/1374527184 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a074ea89-c631-40d6-bb40-703de5636e78", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa27002b020 con 0x7fa27c06b700 2024-01-16T13:51:49.061 INFO:teuthology.orchestra.run.smithi124.stdout:6 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.079+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 msgr2=0x7fa27c0ab810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.079+0000 7fa282b2e700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 0x7fa27c0ab810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa270009fd0 tx=0x7fa270004750 comp rx=0 tx=0).stop 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.079+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 shutdown_connections 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.079+0000 7fa282b2e700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa27c15bd10 0x7fa27c151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.079+0000 7fa282b2e700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa27c06b700 0x7fa27c0ab810 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.079+0000 7fa282b2e700 1 --2- 172.21.15.124:0/1374527184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa27c0af860 0x7fa27c151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.080+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 >> 172.21.15.124:0/1374527184 conn(0x7fa27c0b20c0 msgr2=0x7fa27c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.080+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 shutdown_connections 2024-01-16T13:51:49.063 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.080+0000 7fa282b2e700 1 -- 172.21.15.124:0/1374527184 wait complete. 2024-01-16T13:51:49.076 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 7cda6133-ccbb-4129-ab9b-de80c3f615a0 7 2024-01-16T13:51:49.145 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:49.162+0000 7fa6132c9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T13:51:49.201 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.218+0000 7f288eff2700 1 Processor -- start 2024-01-16T13:51:49.201 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.218+0000 7f288eff2700 1 -- start start 2024-01-16T13:51:49.204 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.219+0000 7f288eff2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28880ccd30 0x7f2888160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.219+0000 7f288eff2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28880b1fe0 0x7f28880b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.219+0000 7f288eff2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288815c9b0 con 0x7f28880afa80 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.219+0000 7f288eff2700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28881607c0 con 0x7f28880ccd30 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.219+0000 7f288eff2700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28880afe60 con 0x7f28880b1fe0 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.219+0000 7f288d58f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28880b1fe0 0x7f28880b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28880b1fe0 0x7f28880b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55892/0 (socket says 172.21.15.124:55892) 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 -- 172.21.15.124:0/2393147878 learned_addr learned my addr 172.21.15.124:0/2393147878 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f2887fff700 1 --2- 172.21.15.124:0/2393147878 >> v2:172.21.15.113:3300/0 conn(0x7f28880ccd30 0x7f2888160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 -- 172.21.15.124:0/2393147878 >> v2:172.21.15.113:3300/0 conn(0x7f28880ccd30 msgr2=0x7f2888160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 --2- 172.21.15.124:0/2393147878 >> v2:172.21.15.113:3300/0 conn(0x7f28880ccd30 0x7f2888160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 -- 172.21.15.124:0/2393147878 >> v1:172.21.15.113:6789/0 conn(0x7f28880afa80 legacy=0x7f288815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 -- 172.21.15.124:0/2393147878 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28880b2900 con 0x7f28880b1fe0 2024-01-16T13:51:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f2887fff700 1 --2- 172.21.15.124:0/2393147878 >> v2:172.21.15.113:3300/0 conn(0x7f28880ccd30 0x7f2888160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f288d58f700 1 --2- 172.21.15.124:0/2393147878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880b1fe0 0x7f28880b23c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28880d0240 tx=0x7f288000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d23896afb98cfa7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f28877fe700 1 -- 172.21.15.124:0/2393147878 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288000e070 con 0x7f28880b1fe0 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f28877fe700 1 -- 172.21.15.124:0/2393147878 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288000b910 con 0x7f28880b1fe0 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.220+0000 7f28877fe700 1 -- 172.21.15.124:0/2393147878 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288000bcc0 con 0x7f28880b1fe0 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 -- 172.21.15.124:0/2393147878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880b1fe0 msgr2=0x7f28880b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 --2- 172.21.15.124:0/2393147878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880b1fe0 0x7f28880b23c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28880d0240 tx=0x7f288000b330 comp rx=0 tx=0).stop 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 -- 172.21.15.124:0/2393147878 shutdown_connections 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 --2- 172.21.15.124:0/2393147878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880b1fe0 0x7f28880b23c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 --2- 172.21.15.124:0/2393147878 >> v2:172.21.15.113:3300/0 conn(0x7f28880ccd30 0x7f2888160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 -- 172.21.15.124:0/2393147878 >> 172.21.15.124:0/2393147878 conn(0x7f2888074170 msgr2=0x7f288806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 -- 172.21.15.124:0/2393147878 shutdown_connections 2024-01-16T13:51:49.206 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.221+0000 7f288eff2700 1 -- 172.21.15.124:0/2393147878 wait complete. 2024-01-16T13:51:49.207 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.223+0000 7f288eff2700 1 Processor -- start 2024-01-16T13:51:49.207 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.223+0000 7f288eff2700 1 -- start start 2024-01-16T13:51:49.209 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288eff2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 0x7f288814ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288eff2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28880b1fe0 0x7f28881f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288eff2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28881f4ff0 0x7f288814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288eff2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880b45a0 con 0x7f28880b1fe0 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288eff2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28881f8850 con 0x7f28880afa80 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288eff2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28881f8290 con 0x7f28881f4ff0 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.224+0000 7f288cd8e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 0x7f288814ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288cd8e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 0x7f288814ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:55896/0 (socket says 172.21.15.124:55896) 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288d58f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28881f4ff0 0x7f288814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288cd8e700 1 -- 172.21.15.124:0/3468874380 learned_addr learned my addr 172.21.15.124:0/3468874380 (peer_addr_for_me v2:172.21.15.124:0/0) 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28880b1fe0 0x7f28881f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 -- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28881f4ff0 msgr2=0x7f288814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28881f4ff0 0x7f288814f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 -- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 msgr2=0x7f288814ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 0x7f288814ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 -- 172.21.15.124:0/3468874380 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28740094a0 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288d58f700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28881f4ff0 0x7f288814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288cd8e700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 0x7f288814ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f2887fff700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28880b1fe0 0x7f28881f79d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f287400e9b0 tx=0x7f287400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287400ee50 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287400c850 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287400cc00 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2880009b30 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28880b4230 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2874005d20 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.225+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f287400cd60 con 0x7f28880b1fe0 2024-01-16T13:51:49.211 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.226+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28880b4710 con 0x7f28880b1fe0 2024-01-16T13:51:49.212 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.229+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2874021060 con 0x7f28880b1fe0 2024-01-16T13:51:49.222 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:49.240+0000 7fa6132c9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T13:51:49.248 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:49.266+0000 7f2693950200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T13:51:49.299 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.316+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7cda6133-ccbb-4129-ab9b-de80c3f615a0", "id": 7} v 0) v1 -- 0x7f2888069020 con 0x7f28880b1fe0 2024-01-16T13:51:49.303 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.320+0000 7f28857fa700 1 -- 172.21.15.124:0/3468874380 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7cda6133-ccbb-4129-ab9b-de80c3f615a0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2874029020 con 0x7f28880b1fe0 2024-01-16T13:51:49.303 INFO:teuthology.orchestra.run.smithi124.stdout:7 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28880b1fe0 msgr2=0x7f28881f79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28880b1fe0 0x7f28881f79d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f287400e9b0 tx=0x7f287400ecc0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 shutdown_connections 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28881f4ff0 0x7f288814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28880afa80 0x7f288814ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 --2- 172.21.15.124:0/3468874380 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28880b1fe0 0x7f28881f79d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 >> 172.21.15.124:0/3468874380 conn(0x7f2888074170 msgr2=0x7f288815f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 shutdown_connections 2024-01-16T13:51:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:2024-01-16T13:51:49.321+0000 7f288eff2700 1 -- 172.21.15.124:0/3468874380 wait complete. 2024-01-16T13:51:49.317 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T13:51:49.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T13:51:49.319 INFO:tasks.ceph.osd.0:Started 2024-01-16T13:51:49.319 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T13:51:49.319 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T13:51:49.321 INFO:tasks.ceph.osd.1:Started 2024-01-16T13:51:49.321 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T13:51:49.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T13:51:49.323 INFO:tasks.ceph.osd.2:Started 2024-01-16T13:51:49.323 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-16T13:51:49.323 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-16T13:51:49.325 INFO:tasks.ceph.osd.3:Started 2024-01-16T13:51:49.326 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-16T13:51:49.326 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-16T13:51:49.328 INFO:tasks.ceph.osd.4:Started 2024-01-16T13:51:49.328 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-16T13:51:49.328 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-16T13:51:49.330 INFO:tasks.ceph.osd.5:Started 2024-01-16T13:51:49.330 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-16T13:51:49.330 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-16T13:51:49.332 INFO:tasks.ceph.osd.6:Started 2024-01-16T13:51:49.332 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-16T13:51:49.332 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-16T13:51:49.334 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:49.350+0000 7f2693950200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T13:51:49.335 INFO:tasks.ceph.osd.7:Started 2024-01-16T13:51:49.336 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:51:49.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.500+0000 7f242f9be700 1 Processor -- start 2024-01-16T13:51:49.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.501+0000 7f242f9be700 1 -- start start 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242f9be700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24281446a0 0x7f24280b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242f9be700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280af950 con 0x7f242806c710 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242f9be700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280afc30 con 0x7f24280adf00 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242f9be700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2428144a80 con 0x7f24281446a0 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242df5b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f242806c710 0x7f24280b3d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:59332/0 (socket says 172.21.15.113:59332) 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242df5b700 1 -- 172.21.15.113:0/3978650239 learned_addr learned my addr 172.21.15.113:0/3978650239 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:51:49.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.502+0000 7f242d75a700 1 --2- 172.21.15.113:0/3978650239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24281446a0 0x7f24280b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235107915 0 0) 0x7f24280af950 con 0x7f242806c710 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24140035b0 con 0x7f242806c710 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013852015 0 0) 0x7f24280afc30 con 0x7f24280adf00 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24280af950 con 0x7f24280adf00 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f242d75a700 1 -- 172.21.15.113:0/3978650239 >> v1:172.21.15.124:6789/0 conn(0x7f24280adf00 legacy=0x7f242806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f242d75a700 1 -- 172.21.15.113:0/3978650239 >> v1:172.21.15.113:6789/0 conn(0x7f242806c710 legacy=0x7f24280b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f242d75a700 1 -- 172.21.15.113:0/3978650239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280aff10 con 0x7f24281446a0 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f242d75a700 1 --2- 172.21.15.113:0/3978650239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24281446a0 0x7f24280b5ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2424009870 tx=0x7f2424009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405b245b3df45bea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424011830 con 0x7f24281446a0 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2424011990 con 0x7f24281446a0 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.503+0000 7f241ffff700 1 -- 172.21.15.113:0/3978650239 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424011d20 con 0x7f24281446a0 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 -- 172.21.15.113:0/3978650239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24281446a0 msgr2=0x7f24280b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 --2- 172.21.15.113:0/3978650239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24281446a0 0x7f24280b5ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2424009870 tx=0x7f2424009b80 comp rx=0 tx=0).stop 2024-01-16T13:51:49.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 -- 172.21.15.113:0/3978650239 shutdown_connections 2024-01-16T13:51:49.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 --2- 172.21.15.113:0/3978650239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24281446a0 0x7f24280b5ae0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 -- 172.21.15.113:0/3978650239 >> 172.21.15.113:0/3978650239 conn(0x7f2428071b90 msgr2=0x7f2428071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:49.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 -- 172.21.15.113:0/3978650239 shutdown_connections 2024-01-16T13:51:49.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.504+0000 7f242f9be700 1 -- 172.21.15.113:0/3978650239 wait complete. 2024-01-16T13:51:49.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.505+0000 7f242f9be700 1 Processor -- start 2024-01-16T13:51:49.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.505+0000 7f242f9be700 1 -- start start 2024-01-16T13:51:49.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242f9be700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 0x7f24280cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242f9be700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24280adf00 0x7f242809c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242f9be700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24281446a0 0x7f24281f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242f9be700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242809c9b0 con 0x7f242806c710 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242f9be700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242809cbf0 con 0x7f24281446a0 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242f9be700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242809ced0 con 0x7f24280adf00 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242d75a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 0x7f24280cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242d75a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 0x7f24280cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54286/0 (socket says 172.21.15.113:54286) 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.506+0000 7f242d75a700 1 -- 172.21.15.113:0/873727606 learned_addr learned my addr 172.21.15.113:0/873727606 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242cf59700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24280adf00 0x7f242809c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242df5b700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24281446a0 0x7f24281f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242d75a700 1 -- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24280adf00 msgr2=0x7f242809c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242d75a700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24280adf00 0x7f242809c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242d75a700 1 -- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24281446a0 msgr2=0x7f24281f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242d75a700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24281446a0 0x7f24281f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242d75a700 1 -- 172.21.15.113:0/873727606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240094a0 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242d75a700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 0x7f24280cffe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f24240090b0 tx=0x7f2424004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242401a6e0 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242401a840 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242809d1b0 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242401abb0 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24281f3f60 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.507+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2424022070 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.508+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f242401ad10 con 0x7f242806c710 2024-01-16T13:51:49.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.508+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24280753f0 con 0x7f242806c710 2024-01-16T13:51:49.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.511+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2424029070 con 0x7f242806c710 2024-01-16T13:51:49.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.582+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f242809d7e0 con 0x7f242806c710 2024-01-16T13:51:49.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.583+0000 7f241dffb700 1 -- 172.21.15.113:0/873727606 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2424015020 con 0x7f242806c710 2024-01-16T13:51:49.566 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:49.566 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":10,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:51:49.318904+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 msgr2=0x7f24280cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 0x7f24280cffe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f24240090b0 tx=0x7f2424004750 comp rx=0 tx=0).stop 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 shutdown_connections 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24281446a0 0x7f24281f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f242806c710 0x7f24280cffe0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 --2- 172.21.15.113:0/873727606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24280adf00 0x7f242809c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 >> 172.21.15.113:0/873727606 conn(0x7f2428071b90 msgr2=0x7f242806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 shutdown_connections 2024-01-16T13:51:49.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:49.584+0000 7f242f9be700 1 -- 172.21.15.113:0/873727606 wait complete. 2024-01-16T13:51:49.578 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T13:51:49.579 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T13:51:49.689 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-16T13:51:49.706+0000 7f44b30a5640 -1 Falling back to public interface 2024-01-16T13:51:49.936 INFO:tasks.ceph.osd.4.smithi124.stderr:2024-01-16T13:51:49.952+0000 7fd1845d8640 -1 Falling back to public interface 2024-01-16T13:51:49.936 INFO:tasks.ceph.osd.6.smithi124.stderr:2024-01-16T13:51:49.953+0000 7f0b0e403640 -1 Falling back to public interface 2024-01-16T13:51:49.936 INFO:tasks.ceph.osd.7.smithi124.stderr:2024-01-16T13:51:49.953+0000 7f0eada68640 -1 Falling back to public interface 2024-01-16T13:51:49.942 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-16T13:51:49.959+0000 7fa7251c1640 -1 Falling back to public interface 2024-01-16T13:51:49.948 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-16T13:51:49.965+0000 7f9c18594640 -1 Falling back to public interface 2024-01-16T13:51:49.952 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-16T13:51:49.969+0000 7f8a4d7e5640 -1 Falling back to public interface 2024-01-16T13:51:49.959 INFO:tasks.ceph.osd.5.smithi124.stderr:2024-01-16T13:51:49.976+0000 7f62cc067640 -1 Falling back to public interface 2024-01-16T13:51:49.974 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:49.991+0000 7fa6132c9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T13:51:50.054 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:50.071+0000 7fa6132c9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T13:51:50.068 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:50.085+0000 7f2693950200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T13:51:50.148 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:50.165+0000 7f2693950200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T13:51:50.184 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:50.202+0000 7fa6132c9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T13:51:50.281 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:50.298+0000 7f2693950200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T13:51:50.384 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T13:51:50.501 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.518+0000 7fb4ad8d8700 1 Processor -- start 2024-01-16T13:51:50.501 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.518+0000 7fb4ad8d8700 1 -- start start 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4ad8d8700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb4a806c710 0x7fb4a80b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4ad8d8700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8144c20 con 0x7fb4a8144840 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4ad8d8700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a815f6a0 con 0x7fb4a815eac0 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4ad8d8700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a80b5a60 con 0x7fb4a806c710 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a6ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb4a8144840 0x7fb4a806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:59338/0 (socket says 172.21.15.113:59338) 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a6ffd700 1 -- 172.21.15.113:0/336576829 learned_addr learned my addr 172.21.15.113:0/336576829 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a67fc700 1 --2- 172.21.15.113:0/336576829 >> v2:172.21.15.124:3300/0 conn(0x7fb4a806c710 0x7fb4a80b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a5ffb700 1 -- 172.21.15.113:0/336576829 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610152506 0 0) 0x7fb4a8144c20 con 0x7fb4a8144840 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a5ffb700 1 -- 172.21.15.113:0/336576829 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4900035b0 con 0x7fb4a8144840 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a67fc700 1 -- 172.21.15.113:0/336576829 >> v1:172.21.15.124:6789/0 conn(0x7fb4a815eac0 legacy=0x7fb4a806db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a67fc700 1 -- 172.21.15.113:0/336576829 >> v1:172.21.15.113:6789/0 conn(0x7fb4a8144840 legacy=0x7fb4a806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.519+0000 7fb4a67fc700 1 -- 172.21.15.113:0/336576829 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a815f980 con 0x7fb4a806c710 2024-01-16T13:51:50.504 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4a67fc700 1 --2- 172.21.15.113:0/336576829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a806c710 0x7fb4a80b47c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb498009840 tx=0x7fb498009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd788e7d75c73ee9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4a5ffb700 1 -- 172.21.15.113:0/336576829 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498011830 con 0x7fb4a806c710 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4a5ffb700 1 -- 172.21.15.113:0/336576829 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb498011990 con 0x7fb4a806c710 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4a5ffb700 1 -- 172.21.15.113:0/336576829 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498011d20 con 0x7fb4a806c710 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/336576829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a806c710 msgr2=0x7fb4a80b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 --2- 172.21.15.113:0/336576829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a806c710 0x7fb4a80b47c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb498009840 tx=0x7fb498009b50 comp rx=0 tx=0).stop 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/336576829 shutdown_connections 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 --2- 172.21.15.113:0/336576829 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a806c710 0x7fb4a80b47c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/336576829 >> 172.21.15.113:0/336576829 conn(0x7fb4a8071ae0 msgr2=0x7fb4a8071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/336576829 shutdown_connections 2024-01-16T13:51:50.505 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.520+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/336576829 wait complete. 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 Processor -- start 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 -- start start 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 0x7fb4a813e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a8144840 0x7fb4a80aa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb4a815eac0 0x7fb4a813f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81f0d90 con 0x7fb4a806c710 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81f1070 con 0x7fb4a8144840 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.522+0000 7fb4ad8d8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81f1350 con 0x7fb4a815eac0 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 0x7fb4a813e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:50.508 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a67fc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a8144840 0x7fb4a80aa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 0x7fb4a813e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54288/0 (socket says 172.21.15.113:54288) 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 -- 172.21.15.113:0/1659891262 learned_addr learned my addr 172.21.15.113:0/1659891262 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a77fe700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb4a815eac0 0x7fb4a813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 -- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb4a815eac0 msgr2=0x7fb4a813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb4a815eac0 0x7fb4a813f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 -- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a8144840 msgr2=0x7fb4a80aa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a8144840 0x7fb4a80aa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 -- 172.21.15.113:0/1659891262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980094a0 con 0x7fb4a806c710 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a67fc700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a8144840 0x7fb4a80aa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4a6ffd700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 0x7fb4a813e4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00c700 tx=0x7fb49c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c005d70 con 0x7fb4a806c710 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c0070d0 con 0x7fb4a806c710 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a81f1630 con 0x7fb4a806c710 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c007460 con 0x7fb4a806c710 2024-01-16T13:51:50.509 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a813ea00 con 0x7fb4a806c710 2024-01-16T13:51:50.510 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.523+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb49c01c440 con 0x7fb4a806c710 2024-01-16T13:51:50.510 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.524+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb49c012ad0 con 0x7fb4a806c710 2024-01-16T13:51:50.510 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.524+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a8068170 con 0x7fb4a806c710 2024-01-16T13:51:50.510 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.526+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb49c017070 con 0x7fb4a806c710 2024-01-16T13:51:50.579 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.596+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4a8068fb0 con 0x7fb4a806c710 2024-01-16T13:51:50.580 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.597+0000 7fb48ffff700 1 -- 172.21.15.113:0/1659891262 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb49c0124b0 con 0x7fb4a806c710 2024-01-16T13:51:50.580 INFO:teuthology.misc.health.smithi113.stdout: 2024-01-16T13:51:50.580 INFO:teuthology.misc.health.smithi113.stdout:{"epoch":10,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:51:49.318904+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 msgr2=0x7fb4a813e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 0x7fb4a813e4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00c700 tx=0x7fb49c00ca10 comp rx=0 tx=0).stop 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 shutdown_connections 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb4a815eac0 0x7fb4a813f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4a806c710 0x7fb4a813e4c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 --2- 172.21.15.113:0/1659891262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4a8144840 0x7fb4a80aa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 >> 172.21.15.113:0/1659891262 conn(0x7fb4a8071ae0 msgr2=0x7fb4a8071ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 shutdown_connections 2024-01-16T13:51:50.581 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:50.598+0000 7fb4ad8d8700 1 -- 172.21.15.113:0/1659891262 wait complete. 2024-01-16T13:51:50.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-16T13:51:50.739 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:50.756+0000 7fa6132c9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T13:51:50.832 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:50.849+0000 7f2693950200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T13:51:50.920 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:50.937+0000 7fa6132c9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T13:51:50.995 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:51.012+0000 7f2693950200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T13:51:51.085 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:51.102+0000 7fa6132c9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T13:51:51.160 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:51.177+0000 7f2693950200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T13:51:51.895 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:51.912+0000 7fa6132c9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T13:51:51.972 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:51.989+0000 7f2693950200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T13:51:52.127 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:52.144+0000 7fa6132c9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T13:51:52.194 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:52.211+0000 7f2693950200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T13:51:52.202 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:52.219+0000 7fa6132c9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T13:51:52.269 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:52.286+0000 7f2693950200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T13:51:52.362 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:52.379+0000 7fa6132c9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T13:51:52.440 INFO:tasks.ceph.mgr.x.smithi124.stderr:2024-01-16T13:51:52.457+0000 7f2693950200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T13:51:53.179 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-16T13:51:53.197+0000 7f44b30a5640 -1 osd.3 0 log_to_monitors true 2024-01-16T13:51:53.386 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:53.403+0000 7fa5ce681700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-16T13:51:53.386 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:51:53.403+0000 7fa5ce681700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-16T13:51:53.437 INFO:tasks.ceph.osd.4.smithi124.stderr:2024-01-16T13:51:53.454+0000 7fd1845d8640 -1 osd.4 0 log_to_monitors true 2024-01-16T13:51:53.458 INFO:tasks.ceph.osd.6.smithi124.stderr:2024-01-16T13:51:53.476+0000 7f0b0e403640 -1 osd.6 0 log_to_monitors true 2024-01-16T13:51:53.460 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-16T13:51:53.477+0000 7f9c18594640 -1 osd.0 0 log_to_monitors true 2024-01-16T13:51:53.461 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-16T13:51:53.478+0000 7fa7251c1640 -1 osd.1 0 log_to_monitors true 2024-01-16T13:51:53.469 INFO:tasks.ceph.osd.7.smithi124.stderr:2024-01-16T13:51:53.487+0000 7f0eada68640 -1 osd.7 0 log_to_monitors true 2024-01-16T13:51:53.475 INFO:tasks.ceph.osd.5.smithi124.stderr:2024-01-16T13:51:53.492+0000 7f62cc067640 -1 osd.5 0 log_to_monitors true 2024-01-16T13:51:53.476 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-16T13:51:53.493+0000 7f8a4d7e5640 -1 osd.2 0 log_to_monitors true 2024-01-16T13:51:55.934 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-16T13:51:55.951+0000 7fa718d75700 -1 osd.1 0 waiting for initial osdmap 2024-01-16T13:51:55.948 INFO:tasks.ceph.osd.1.smithi113.stderr:2024-01-16T13:51:55.965+0000 7fa711155700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:55.953 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-16T13:51:55.970+0000 7f8a41399700 -1 osd.2 0 waiting for initial osdmap 2024-01-16T13:51:55.963 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-16T13:51:55.980+0000 7f9c0c148700 -1 osd.0 0 waiting for initial osdmap 2024-01-16T13:51:55.963 INFO:tasks.ceph.osd.2.smithi113.stderr:2024-01-16T13:51:55.981+0000 7f8a39779700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:55.976 INFO:tasks.ceph.osd.0.smithi113.stderr:2024-01-16T13:51:55.994+0000 7f9c04528700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:55.980 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-16T13:51:55.997+0000 7f44a6c59700 -1 osd.3 0 waiting for initial osdmap 2024-01-16T13:51:55.992 INFO:tasks.ceph.osd.3.smithi113.stderr:2024-01-16T13:51:56.009+0000 7f449f039700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:56.509 INFO:tasks.ceph.osd.7.smithi124.stderr:2024-01-16T13:51:56.526+0000 7f0ea161c700 -1 osd.7 0 waiting for initial osdmap 2024-01-16T13:51:56.527 INFO:tasks.ceph.osd.7.smithi124.stderr:2024-01-16T13:51:56.544+0000 7f0e999fc700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:56.572 INFO:tasks.ceph.osd.4.smithi124.stderr:2024-01-16T13:51:56.590+0000 7fd17818c700 -1 osd.4 0 waiting for initial osdmap 2024-01-16T13:51:56.584 INFO:tasks.ceph.osd.4.smithi124.stderr:2024-01-16T13:51:56.601+0000 7fd17056c700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:56.596 INFO:tasks.ceph.osd.5.smithi124.stderr:2024-01-16T13:51:56.614+0000 7f62bfc1b700 -1 osd.5 0 waiting for initial osdmap 2024-01-16T13:51:56.601 INFO:tasks.ceph.osd.6.smithi124.stderr:2024-01-16T13:51:56.618+0000 7f0b01fb7700 -1 osd.6 0 waiting for initial osdmap 2024-01-16T13:51:56.614 INFO:tasks.ceph.osd.5.smithi124.stderr:2024-01-16T13:51:56.631+0000 7f62b7ffb700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:56.614 INFO:tasks.ceph.osd.6.smithi124.stderr:2024-01-16T13:51:56.631+0000 7f0afa397700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T13:51:57.398 DEBUG:teuthology.orchestra.run.smithi113:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T13:51:57.524 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.541+0000 7f5aa2167700 1 Processor -- start 2024-01-16T13:51:57.524 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.541+0000 7f5aa2167700 1 -- start start 2024-01-16T13:51:57.526 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5aa2167700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147660 0x7f5a9c15f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.526 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5aa2167700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5a9c06c5d0 0x7f5a9c0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.526 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5aa2167700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9c15f680 con 0x7f5a9c147a40 2024-01-16T13:51:57.526 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5aa2167700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9c15e680 con 0x7f5a9c06c5d0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5aa2167700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c0d05e0 con 0x7f5a9c147660 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5a9affd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5a9c06c5d0 0x7f5a9c0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5a9bfff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f5a9c147a40 0x7f5a9c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:57152/0 (socket says 172.21.15.113:57152) 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5a9bfff700 1 -- 172.21.15.113:0/1799372670 learned_addr learned my addr 172.21.15.113:0/1799372670 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.542+0000 7f5a9b7fe700 1 --2- 172.21.15.113:0/1799372670 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147660 0x7f5a9c15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9affd700 1 -- 172.21.15.113:0/1799372670 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147660 msgr2=0x7f5a9c15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9affd700 1 --2- 172.21.15.113:0/1799372670 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147660 0x7f5a9c15f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9affd700 1 -- 172.21.15.113:0/1799372670 >> v1:172.21.15.113:6789/0 conn(0x7f5a9c147a40 legacy=0x7f5a9c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9a7fc700 1 -- 172.21.15.113:0/1799372670 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372343415 0 0) 0x7f5a94001490 con 0x7f5a9c147a40 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9affd700 1 -- 172.21.15.113:0/1799372670 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c0d08c0 con 0x7f5a9c06c5d0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9affd700 1 --2- 172.21.15.113:0/1799372670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c0ce800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5a90009840 tx=0x7f5a90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e9121ef1a881c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9a7fc700 1 -- 172.21.15.113:0/1799372670 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a90011830 con 0x7f5a9c06c5d0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9a7fc700 1 -- 172.21.15.113:0/1799372670 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a90011990 con 0x7f5a9c06c5d0 2024-01-16T13:51:57.527 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.543+0000 7f5a9a7fc700 1 -- 172.21.15.113:0/1799372670 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a90011d20 con 0x7f5a9c06c5d0 2024-01-16T13:51:57.528 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 -- 172.21.15.113:0/1799372670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 msgr2=0x7f5a9c0ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.528 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1799372670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c0ce800 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5a90009840 tx=0x7f5a90009b50 comp rx=0 tx=0).stop 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 -- 172.21.15.113:0/1799372670 shutdown_connections 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1799372670 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147660 0x7f5a9c15f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1799372670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c0ce800 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 -- 172.21.15.113:0/1799372670 >> 172.21.15.113:0/1799372670 conn(0x7f5a9c0b2940 msgr2=0x7f5a9c0b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 -- 172.21.15.113:0/1799372670 shutdown_connections 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.544+0000 7f5aa2167700 1 -- 172.21.15.113:0/1799372670 wait complete. 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.545+0000 7f5aa2167700 1 Processor -- start 2024-01-16T13:51:57.529 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 -- start start 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c13d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 0x7f5a9c145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147a40 0x7f5a9c1461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1f4d40 con 0x7f5a9c06c5d0 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1f3f80 con 0x7f5a9c147660 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.546+0000 7f5aa2167700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9c1f4260 con 0x7f5a9c147a40 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 0x7f5a9c145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9bfff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147a40 0x7f5a9c1461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 0x7f5a9c145ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:49408/0 (socket says 172.21.15.113:49408) 2024-01-16T13:51:57.531 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 -- 172.21.15.113:0/1029396847 learned_addr learned my addr 172.21.15.113:0/1029396847 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9b7fe700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c13d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 -- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147a40 msgr2=0x7f5a9c1461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147a40 0x7f5a9c1461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 -- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 msgr2=0x7f5a9c13d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c13d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 -- 172.21.15.113:0/1029396847 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a900094a0 con 0x7f5a9c147660 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a9affd700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 0x7f5a9c145ca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a90009fd0 tx=0x7f5a90004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a900036a0 con 0x7f5a9c147660 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9001f070 con 0x7f5a9c147660 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c1f4540 con 0x7f5a9c147660 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a90003a60 con 0x7f5a9c147660 2024-01-16T13:51:57.532 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.547+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c0b4600 con 0x7f5a9c147660 2024-01-16T13:51:57.533 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.549+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9c1f4a60 con 0x7f5a9c147660 2024-01-16T13:51:57.533 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.549+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f5a90003bc0 con 0x7f5a9c147660 2024-01-16T13:51:57.533 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.549+0000 7f5a83fff700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5a8806b2c0 0x7f5a8806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.533 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.549+0000 7f5a9b7fe700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5a8806b2c0 0x7f5a8806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.533 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.550+0000 7f5a9b7fe700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5a8806b2c0 0x7f5a8806d780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c15b880 tx=0x7f5a8c009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:57.534 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.550+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f5a90021070 con 0x7f5a9c147660 2024-01-16T13:51:57.538 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.555+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a9c1f4a60 con 0x7f5a9c147660 2024-01-16T13:51:57.732 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.749+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a9c1f4a60 con 0x7f5a9c147660 2024-01-16T13:51:57.733 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.750+0000 7f5a83fff700 1 -- 172.21.15.113:0/1029396847 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+11128 (secure 0 0 0) 0x7f5a9c1f4a60 con 0x7f5a9c147660 2024-01-16T13:51:57.733 INFO:teuthology.misc.health.smithi113.stdout: 2024-01-16T13:51:57.733 INFO:teuthology.misc.health.smithi113.stdout:{"epoch":14,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:51:57.673574+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:51:57.735 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5a8806b2c0 msgr2=0x7f5a8806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5a8806b2c0 0x7f5a8806d780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c15b880 tx=0x7f5a8c009210 comp rx=0 tx=0).stop 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 msgr2=0x7f5a9c145ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 0x7f5a9c145ca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a90009fd0 tx=0x7f5a90004750 comp rx=0 tx=0).stop 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 shutdown_connections 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5a8806b2c0 0x7f5a8806d780 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a9c147a40 0x7f5a9c1461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a9c06c5d0 0x7f5a9c13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 --2- 172.21.15.113:0/1029396847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a9c147660 0x7f5a9c145ca0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 >> 172.21.15.113:0/1029396847 conn(0x7f5a9c0b2940 msgr2=0x7f5a9c161bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 shutdown_connections 2024-01-16T13:51:57.736 INFO:teuthology.misc.health.smithi113.stderr:2024-01-16T13:51:57.752+0000 7f5aa2167700 1 -- 172.21.15.113:0/1029396847 wait complete. 2024-01-16T13:51:57.746 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-16T13:51:57.747 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T13:51:57.747 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T13:51:57.747 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T13:51:57.749 INFO:tasks.ceph.mds.a:Started 2024-01-16T13:51:57.749 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T13:51:57.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T13:51:57.751 INFO:tasks.ceph.mds.c:Started 2024-01-16T13:51:57.751 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T13:51:57.751 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T13:51:57.754 INFO:tasks.ceph.mds.b:Started 2024-01-16T13:51:57.754 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T13:51:57.754 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T13:51:57.756 INFO:tasks.ceph.mds.d:Started 2024-01-16T13:51:57.756 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-16T13:51:57.756 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:51:57.830 INFO:tasks.ceph.mds.c.smithi113.stdout:starting mds.c at 2024-01-16T13:51:57.830 INFO:tasks.ceph.mds.a.smithi113.stdout:starting mds.a at 2024-01-16T13:51:57.846 INFO:tasks.ceph.mds.b.smithi124.stdout:starting mds.b at 2024-01-16T13:51:57.846 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2024-01-16T13:51:57.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.900+0000 7f0b03d3a700 1 Processor -- start 2024-01-16T13:51:57.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 -- start start 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0afc15e080 0x7f0afc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc0af4d0 0x7f0afc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0afc06c000 0x7f0afc06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0afc0af0b0 con 0x7f0afc06c000 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0afc06c9f0 con 0x7f0afc15e080 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.901+0000 7f0b03d3a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0b05a0 con 0x7f0afc0af4d0 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0afc06c000 0x7f0afc06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0afc06c000 0x7f0afc06c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33822/0 (socket says 172.21.15.113:33822) 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 -- 172.21.15.113:0/1231442520 learned_addr learned my addr 172.21.15.113:0/1231442520 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b012d5700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc0af4d0 0x7f0afc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b01ad6700 1 --2- 172.21.15.113:0/1231442520 >> v2:172.21.15.124:3300/0 conn(0x7f0afc15e080 0x7f0afc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 -- 172.21.15.113:0/1231442520 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc0af4d0 msgr2=0x7f0afc0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:57.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc0af4d0 0x7f0afc0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 -- 172.21.15.113:0/1231442520 >> v2:172.21.15.124:3300/0 conn(0x7f0afc15e080 msgr2=0x7f0afc0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 --2- 172.21.15.113:0/1231442520 >> v2:172.21.15.124:3300/0 conn(0x7f0afc15e080 0x7f0afc0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 -- 172.21.15.113:0/1231442520 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc0ae430 con 0x7f0afc06c000 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b012d5700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc0af4d0 0x7f0afc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b022d7700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc06c000 0x7f0afc06c4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0afc072a90 tx=0x7f0af400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5c662004f9ae15 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b00ad4700 1 -- 172.21.15.113:0/1231442520 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af400e070 con 0x7f0afc06c000 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b00ad4700 1 -- 172.21.15.113:0/1231442520 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af400b980 con 0x7f0afc06c000 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.902+0000 7f0b00ad4700 1 -- 172.21.15.113:0/1231442520 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af400bd30 con 0x7f0afc06c000 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 -- 172.21.15.113:0/1231442520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc06c000 msgr2=0x7f0afc06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc06c000 0x7f0afc06c4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0afc072a90 tx=0x7f0af400b330 comp rx=0 tx=0).stop 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 -- 172.21.15.113:0/1231442520 shutdown_connections 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc06c000 0x7f0afc06c4b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/1231442520 >> v2:172.21.15.124:3300/0 conn(0x7f0afc15e080 0x7f0afc0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/1231442520 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc0af4d0 0x7f0afc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 -- 172.21.15.113:0/1231442520 >> 172.21.15.113:0/1231442520 conn(0x7f0afc071b90 msgr2=0x7f0afc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:57.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 -- 172.21.15.113:0/1231442520 shutdown_connections 2024-01-16T13:51:57.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.903+0000 7f0b03d3a700 1 -- 172.21.15.113:0/1231442520 wait complete. 2024-01-16T13:51:57.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.904+0000 7f0b03d3a700 1 Processor -- start 2024-01-16T13:51:57.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.904+0000 7f0b03d3a700 1 -- start start 2024-01-16T13:51:57.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc06c000 0x7f0afc148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 0x7f0afc156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0afc15e080 0x7f0afc0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc1493d0 con 0x7f0afc0af4d0 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc15e840 con 0x7f0afc15e080 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc15eb20 con 0x7f0afc06c000 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 0x7f0afc156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b01ad6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc06c000 0x7f0afc148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b022d7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0afc15e080 0x7f0afc0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b01ad6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc06c000 0x7f0afc148e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:43738/0 (socket says 172.21.15.113:43738) 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 0x7f0afc156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33836/0 (socket says 172.21.15.113:33836) 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b01ad6700 1 -- 172.21.15.113:0/3769228553 learned_addr learned my addr 172.21.15.113:0/3769228553 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 -- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc06c000 msgr2=0x7f0afc148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc06c000 0x7f0afc148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 -- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0afc15e080 msgr2=0x7f0afc0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0afc15e080 0x7f0afc0cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 -- 172.21.15.113:0/3769228553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4009b30 con 0x7f0afc0af4d0 2024-01-16T13:51:57.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b022d7700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0afc15e080 0x7f0afc0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b012d5700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 0x7f0afc156b70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c7c0 tx=0x7f0af800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8004230 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af8004390 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0afc15ee00 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af80036a0 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.905+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc15f3c0 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.906+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0afc15f940 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.907+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f0af8003800 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.907+0000 7f0af27fc700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0ae006b2c0 0x7f0ae006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.907+0000 7f0b01ad6700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0ae006b2c0 0x7f0ae006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.907+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f0af8089270 con 0x7f0afc0af4d0 2024-01-16T13:51:57.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.908+0000 7f0b01ad6700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0ae006b2c0 0x7f0ae006d780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0aec007af0 tx=0x7f0aec008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:57.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:57.912+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0af80545f0 con 0x7f0afc0af4d0 2024-01-16T13:51:58.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.102+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0afc0753f0 con 0x7f0afc0af4d0 2024-01-16T13:51:58.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.103+0000 7f0af27fc700 1 -- 172.21.15.113:0/3769228553 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+11128 (secure 0 0 0) 0x7f0af8057c10 con 0x7f0afc0af4d0 2024-01-16T13:51:58.086 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:58.086 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":14,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:51:57.673574+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0ae006b2c0 msgr2=0x7f0ae006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0ae006b2c0 0x7f0ae006d780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0aec007af0 tx=0x7f0aec008040 comp rx=0 tx=0).stop 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 msgr2=0x7f0afc156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 0x7f0afc156b70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c7c0 tx=0x7f0af800cad0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 shutdown_connections 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0ae006b2c0 0x7f0ae006d780 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0afc15e080 0x7f0afc0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0afc06c000 0x7f0afc148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 --2- 172.21.15.113:0/3769228553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0afc0af4d0 0x7f0afc156b70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 >> 172.21.15.113:0/3769228553 conn(0x7f0afc071b90 msgr2=0x7f0afc073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 shutdown_connections 2024-01-16T13:51:58.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.105+0000 7f0b03d3a700 1 -- 172.21.15.113:0/3769228553 wait complete. 2024-01-16T13:51:58.100 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-01-16T13:51:58.101 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:51:58.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.239+0000 7fb527b47700 1 Processor -- start 2024-01-16T13:51:58.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.239+0000 7fb527b47700 1 -- start start 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb527b47700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb52015b200 0x7fb5200b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb527b47700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb5200b1a70 0x7fb520164b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb527b47700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5200cfd40 con 0x7fb52015ae20 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb527b47700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200d0020 con 0x7fb5200b1a70 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb527b47700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200affa0 con 0x7fb52015b200 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb5260e4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb5200b1a70 0x7fb520164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb5260e4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb5200b1a70 0x7fb520164b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33844/0 (socket says 172.21.15.113:33844) 2024-01-16T13:51:58.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb5260e4700 1 -- 172.21.15.113:0/955123668 learned_addr learned my addr 172.21.15.113:0/955123668 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.240+0000 7fb5250e2700 1 --2- 172.21.15.113:0/955123668 >> v2:172.21.15.124:3300/0 conn(0x7fb52015b200 0x7fb5200b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5260e4700 1 -- 172.21.15.113:0/955123668 >> v2:172.21.15.124:3300/0 conn(0x7fb52015b200 msgr2=0x7fb5200b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5260e4700 1 --2- 172.21.15.113:0/955123668 >> v2:172.21.15.124:3300/0 conn(0x7fb52015b200 0x7fb5200b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5260e4700 1 -- 172.21.15.113:0/955123668 >> v1:172.21.15.124:6789/0 conn(0x7fb52015ae20 legacy=0x7fb5200b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5260e4700 1 -- 172.21.15.113:0/955123668 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5200b0280 con 0x7fb5200b1a70 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5260e4700 1 --2- 172.21.15.113:0/955123668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb520164b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5200cefe0 tx=0x7fb51800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2497b216b3fba6d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5248e1700 1 -- 172.21.15.113:0/955123668 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51800e070 con 0x7fb5200b1a70 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5248e1700 1 -- 172.21.15.113:0/955123668 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51800b910 con 0x7fb5200b1a70 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb5248e1700 1 -- 172.21.15.113:0/955123668 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51800bcc0 con 0x7fb5200b1a70 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb527b47700 1 -- 172.21.15.113:0/955123668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 msgr2=0x7fb520164b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.241+0000 7fb527b47700 1 --2- 172.21.15.113:0/955123668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb520164b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5200cefe0 tx=0x7fb51800b330 comp rx=0 tx=0).stop 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.242+0000 7fb527b47700 1 -- 172.21.15.113:0/955123668 shutdown_connections 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.242+0000 7fb527b47700 1 --2- 172.21.15.113:0/955123668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb520164b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.242+0000 7fb527b47700 1 --2- 172.21.15.113:0/955123668 >> v2:172.21.15.124:3300/0 conn(0x7fb52015b200 0x7fb5200b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.242+0000 7fb527b47700 1 -- 172.21.15.113:0/955123668 >> 172.21.15.113:0/955123668 conn(0x7fb5200b57b0 msgr2=0x7fb5200b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:58.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.242+0000 7fb527b47700 1 -- 172.21.15.113:0/955123668 shutdown_connections 2024-01-16T13:51:58.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.242+0000 7fb527b47700 1 -- 172.21.15.113:0/955123668 wait complete. 2024-01-16T13:51:58.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.243+0000 7fb527b47700 1 Processor -- start 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.243+0000 7fb527b47700 1 -- start start 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb52006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb52015ae20 0x7fb5200cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb52015b200 0x7fb520154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200cb310 con 0x7fb5200b1a70 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52006e030 con 0x7fb52015b200 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52006e310 con 0x7fb52015ae20 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb52006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb52006d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33858/0 (socket says 172.21.15.113:33858) 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 -- 172.21.15.113:0/2827736544 learned_addr learned my addr 172.21.15.113:0/2827736544 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5260e4700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb52015b200 0x7fb520154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5250e2700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb52015ae20 0x7fb5200cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 -- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb52015ae20 msgr2=0x7fb5200cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb52015ae20 0x7fb5200cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 -- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb52015b200 msgr2=0x7fb520154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb52015b200 0x7fb520154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 -- 172.21.15.113:0/2827736544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb518009b30 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5258e3700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb52006d930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c4e0 tx=0x7fb51000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5100052d0 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb510005430 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52006e5f0 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5100038c0 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.244+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb520154cc0 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.245+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5201f3e90 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.246+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fb510003a20 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.246+0000 7fb5167fc700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb50406b2c0 0x7fb50406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.246+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fb510088970 con 0x7fb5200b1a70 2024-01-16T13:51:58.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.246+0000 7fb5250e2700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb50406b2c0 0x7fb50406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:51:58.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.246+0000 7fb5250e2700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb50406b2c0 0x7fb50406d780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb5200cb490 tx=0x7fb51c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:51:58.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.251+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5201f3e90 con 0x7fb5200b1a70 2024-01-16T13:51:58.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.485+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb520067f60 con 0x7fb5200b1a70 2024-01-16T13:51:58.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.486+0000 7fb5167fc700 1 -- 172.21.15.113:0/2827736544 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7fb510017090 con 0x7fb5200b1a70 2024-01-16T13:51:58.469 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:51:58.469 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-16T13:51:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb50406b2c0 msgr2=0x7fb50406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb50406b2c0 0x7fb50406d780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb5200cb490 tx=0x7fb51c008040 comp rx=0 tx=0).stop 2024-01-16T13:51:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 msgr2=0x7fb52006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:51:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb52006d930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c4e0 tx=0x7fb51000c7f0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 shutdown_connections 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb50406b2c0 0x7fb50406d780 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb52015b200 0x7fb520154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5200b1a70 0x7fb52006d930 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 --2- 172.21.15.113:0/2827736544 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb52015ae20 0x7fb5200cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 >> 172.21.15.113:0/2827736544 conn(0x7fb5200b57b0 msgr2=0x7fb5200d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 shutdown_connections 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:51:58.488+0000 7fb527b47700 1 -- 172.21.15.113:0/2827736544 wait complete. 2024-01-16T13:51:58.472 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 2 2024-01-16T13:52:00.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:00.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.641+0000 7f6a3e7c0700 1 Processor -- start 2024-01-16T13:52:00.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.641+0000 7f6a3e7c0700 1 -- start start 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3e7c0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a38072df0 0x7f6a380af070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3e7c0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6a380622a0 0x7f6a380af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3e7c0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380adfd0 con 0x7f6a3815c140 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3e7c0700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380ae2b0 con 0x7f6a380622a0 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3e7c0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380ae590 con 0x7f6a38072df0 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a37fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6a3815c140 0x7f6a380726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:57162/0 (socket says 172.21.15.113:57162) 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a37fff700 1 -- 172.21.15.113:0/517553662 learned_addr learned my addr 172.21.15.113:0/517553662 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3cd5d700 1 --2- 172.21.15.113:0/517553662 >> v2:172.21.15.113:3300/0 conn(0x7f6a380622a0 0x7f6a380af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a377fe700 1 --2- 172.21.15.113:0/517553662 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a38072df0 0x7f6a380af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a36ffd700 1 -- 172.21.15.113:0/517553662 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800236844 0 0) 0x7f6a380adfd0 con 0x7f6a3815c140 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a36ffd700 1 -- 172.21.15.113:0/517553662 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a200035b0 con 0x7f6a3815c140 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3cd5d700 1 -- 172.21.15.113:0/517553662 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a38072df0 msgr2=0x7f6a380af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:00.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3cd5d700 1 --2- 172.21.15.113:0/517553662 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a38072df0 0x7f6a380af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3cd5d700 1 -- 172.21.15.113:0/517553662 >> v1:172.21.15.113:6789/0 conn(0x7f6a3815c140 legacy=0x7f6a380726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3cd5d700 1 -- 172.21.15.113:0/517553662 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380ae870 con 0x7f6a380622a0 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a3cd5d700 1 --2- 172.21.15.113:0/517553662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380622a0 0x7f6a380af5b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009d90 tx=0x7f6a3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf2d72b424ff205 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a36ffd700 1 -- 172.21.15.113:0/517553662 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000e070 con 0x7f6a380622a0 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a36ffd700 1 -- 172.21.15.113:0/517553662 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3000bb20 con 0x7f6a380622a0 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.642+0000 7f6a36ffd700 1 -- 172.21.15.113:0/517553662 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30012430 con 0x7f6a380622a0 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/517553662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380622a0 msgr2=0x7f6a380af5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/517553662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380622a0 0x7f6a380af5b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009d90 tx=0x7f6a3000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/517553662 shutdown_connections 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/517553662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380622a0 0x7f6a380af5b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/517553662 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a38072df0 0x7f6a380af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/517553662 >> 172.21.15.113:0/517553662 conn(0x7f6a380b57b0 msgr2=0x7f6a380b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/517553662 shutdown_connections 2024-01-16T13:52:00.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.643+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/517553662 wait complete. 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.644+0000 7f6a3e7c0700 1 Processor -- start 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.644+0000 7f6a3e7c0700 1 -- start start 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3e7c0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380622a0 0x7f6a380c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3e7c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 0x7f6a3814d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3e7c0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a3815c140 0x7f6a380b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3e7c0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38152cc0 con 0x7f6a38072df0 2024-01-16T13:52:00.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3e7c0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380b35a0 con 0x7f6a380622a0 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3e7c0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380cba80 con 0x7f6a3815c140 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 0x7f6a3814d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 0x7f6a3814d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33890/0 (socket says 172.21.15.113:33890) 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a377fe700 1 -- 172.21.15.113:0/1612121804 learned_addr learned my addr 172.21.15.113:0/1612121804 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a3cd5d700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a3815c140 0x7f6a380b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.645+0000 7f6a37fff700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380622a0 0x7f6a380c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a377fe700 1 -- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a3815c140 msgr2=0x7f6a380b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a377fe700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a3815c140 0x7f6a380b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a377fe700 1 -- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380622a0 msgr2=0x7f6a380c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a377fe700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380622a0 0x7f6a380c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a377fe700 1 -- 172.21.15.113:0/1612121804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30009a70 con 0x7f6a38072df0 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a377fe700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 0x7f6a3814d9a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00e9e0 tx=0x7f6a2c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:00.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c009710 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c00ee80 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a380ca290 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00ca50 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.646+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a380cb4c0 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.647+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a38067f60 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.647+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f6a2c00cbb0 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.648+0000 7f6a34ff9700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b2c0 0x7f6a2406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.648+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a2c0896a0 con 0x7f6a38072df0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.648+0000 7f6a37fff700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b2c0 0x7f6a2406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:00.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.648+0000 7f6a37fff700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b2c0 0x7f6a2406d780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a28007fd0 tx=0x7f6a28007dc0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:00.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.653+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a2c054940 con 0x7f6a38072df0 2024-01-16T13:52:00.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.908+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a380b3390 con 0x7f6a38072df0 2024-01-16T13:52:00.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.909+0000 7f6a34ff9700 1 -- 172.21.15.113:0/1612121804 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3703 (secure 0 0 0) 0x7f6a2c021e90 con 0x7f6a38072df0 2024-01-16T13:52:00.892 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:00.892 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-16T13:52:00.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b2c0 msgr2=0x7f6a2406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:00.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b2c0 0x7f6a2406d780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a28007fd0 tx=0x7f6a28007dc0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 msgr2=0x7f6a3814d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:00.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 0x7f6a3814d9a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00e9e0 tx=0x7f6a2c00ecf0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 shutdown_connections 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b2c0 0x7f6a2406d780 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a3815c140 0x7f6a380b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380622a0 0x7f6a380c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 --2- 172.21.15.113:0/1612121804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a38072df0 0x7f6a3814d9a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 >> 172.21.15.113:0/1612121804 conn(0x7f6a380b57b0 msgr2=0x7f6a380b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 shutdown_connections 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:00.911+0000 7f6a3e7c0700 1 -- 172.21.15.113:0/1612121804 wait complete. 2024-01-16T13:52:00.895 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 3 2024-01-16T13:52:00.906 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:01.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.045+0000 7f2ce3829700 1 Processor -- start 2024-01-16T13:52:01.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.045+0000 7f2ce3829700 1 -- start start 2024-01-16T13:52:01.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce3829700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2cdc15e080 0x7f2cdc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce3829700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc06c710 0x7f2cdc0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce3829700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0ae430 con 0x7f2cdc0af4d0 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce3829700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc0af0b0 con 0x7f2cdc15e080 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce3829700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0b05a0 con 0x7f2cdc06c710 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce0dc4700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2cdc0af4d0 0x7f2cdc06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:57166/0 (socket says 172.21.15.113:57166) 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce0dc4700 1 -- 172.21.15.113:0/2766446664 learned_addr learned my addr 172.21.15.113:0/2766446664 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce15c5700 1 --2- 172.21.15.113:0/2766446664 >> v2:172.21.15.113:3300/0 conn(0x7f2cdc15e080 0x7f2cdc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce1dc6700 1 --2- 172.21.15.113:0/2766446664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc06c710 0x7f2cdc0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2cd3fff700 1 -- 172.21.15.113:0/2766446664 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47450838 0 0) 0x7f2cdc0ae430 con 0x7f2cdc0af4d0 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2cd3fff700 1 -- 172.21.15.113:0/2766446664 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc80035b0 con 0x7f2cdc0af4d0 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce15c5700 1 -- 172.21.15.113:0/2766446664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc06c710 msgr2=0x7f2cdc0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce15c5700 1 --2- 172.21.15.113:0/2766446664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc06c710 0x7f2cdc0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce15c5700 1 -- 172.21.15.113:0/2766446664 >> v1:172.21.15.113:6789/0 conn(0x7f2cdc0af4d0 legacy=0x7f2cdc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce15c5700 1 -- 172.21.15.113:0/2766446664 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0ae710 con 0x7f2cdc15e080 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2ce15c5700 1 --2- 172.21.15.113:0/2766446664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc0b5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009870 tx=0x7f2cd8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e2b21e6ab2800 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2cd3fff700 1 -- 172.21.15.113:0/2766446664 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8011830 con 0x7f2cdc15e080 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2cd3fff700 1 -- 172.21.15.113:0/2766446664 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8011990 con 0x7f2cdc15e080 2024-01-16T13:52:01.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.046+0000 7f2cd3fff700 1 -- 172.21.15.113:0/2766446664 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8011d20 con 0x7f2cdc15e080 2024-01-16T13:52:01.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 -- 172.21.15.113:0/2766446664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 msgr2=0x7f2cdc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 --2- 172.21.15.113:0/2766446664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc0b5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009870 tx=0x7f2cd8009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 -- 172.21.15.113:0/2766446664 shutdown_connections 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 --2- 172.21.15.113:0/2766446664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc06c710 0x7f2cdc0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 --2- 172.21.15.113:0/2766446664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc0b5050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 -- 172.21.15.113:0/2766446664 >> 172.21.15.113:0/2766446664 conn(0x7f2cdc071b90 msgr2=0x7f2cdc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 -- 172.21.15.113:0/2766446664 shutdown_connections 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.047+0000 7f2ce3829700 1 -- 172.21.15.113:0/2766446664 wait complete. 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.049+0000 7f2ce3829700 1 Processor -- start 2024-01-16T13:52:01.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.049+0000 7f2ce3829700 1 -- start start 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2cdc06c710 0x7f2cdc1f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc0af4d0 0x7f2cdc146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc1f7e50 con 0x7f2cdc15e080 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc158220 con 0x7f2cdc06c710 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc158500 con 0x7f2cdc0af4d0 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33910/0 (socket says 172.21.15.113:33910) 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 -- 172.21.15.113:0/1206578860 learned_addr learned my addr 172.21.15.113:0/1206578860 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce0dc4700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc0af4d0 0x7f2cdc146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce15c5700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2cdc06c710 0x7f2cdc1f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 -- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc0af4d0 msgr2=0x7f2cdc146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc0af4d0 0x7f2cdc146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 -- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2cdc06c710 msgr2=0x7f2cdc1f7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2cdc06c710 0x7f2cdc1f7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 -- 172.21.15.113:0/1206578860 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80094a0 con 0x7f2cdc15e080 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce1dc6700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc154830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400efc0 tx=0x7f2cd400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd400cc90 con 0x7f2cdc15e080 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd400f040 con 0x7f2cdc15e080 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc1587e0 con 0x7f2cdc15e080 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.050+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40146c0 con 0x7f2cdc15e080 2024-01-16T13:52:01.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.051+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc146990 con 0x7f2cdc15e080 2024-01-16T13:52:01.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.052+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f2cd4014820 con 0x7f2cdc15e080 2024-01-16T13:52:01.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.052+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc0753f0 con 0x7f2cdc15e080 2024-01-16T13:52:01.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.052+0000 7f2cd1ffb700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2cc006b2c0 0x7f2cc006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.053+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2cd408a1f0 con 0x7f2cdc15e080 2024-01-16T13:52:01.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.053+0000 7f2ce15c5700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2cc006b2c0 0x7f2cc006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.053+0000 7f2ce15c5700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2cc006b2c0 0x7f2cc006d780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80090b0 tx=0x7f2cd801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.057+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cd4055520 con 0x7f2cdc15e080 2024-01-16T13:52:01.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.249+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cdc0d09b0 con 0x7f2cdc15e080 2024-01-16T13:52:01.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.250+0000 7f2cd1ffb700 1 -- 172.21.15.113:0/1206578860 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12886 (secure 0 0 0) 0x7f2cd4009ca0 con 0x7f2cdc15e080 2024-01-16T13:52:01.233 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:01.233 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":17,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:00.860475+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:01.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2cc006b2c0 msgr2=0x7f2cc006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2cc006b2c0 0x7f2cc006d780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80090b0 tx=0x7f2cd801c040 comp rx=0 tx=0).stop 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 msgr2=0x7f2cdc154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc154830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400efc0 tx=0x7f2cd400c3a0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 shutdown_connections 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2cc006b2c0 0x7f2cc006d780 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc15e080 0x7f2cdc154830 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2cdc06c710 0x7f2cdc1f7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 --2- 172.21.15.113:0/1206578860 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2cdc0af4d0 0x7f2cdc146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 >> 172.21.15.113:0/1206578860 conn(0x7f2cdc071b90 msgr2=0x7f2cdc06ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 shutdown_connections 2024-01-16T13:52:01.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.252+0000 7f2ce3829700 1 -- 172.21.15.113:0/1206578860 wait complete. 2024-01-16T13:52:01.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:01.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:01.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.393+0000 7fa74cc24700 1 Processor -- start 2024-01-16T13:52:01.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.393+0000 7fa74cc24700 1 -- start start 2024-01-16T13:52:01.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74cc24700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa74815e110 0x7fa74806f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74cc24700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7480afb30 0x7fa74806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74cc24700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74806d980 con 0x7fa74806b530 2024-01-16T13:52:01.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74cc24700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7480affb0 con 0x7fa74815e110 2024-01-16T13:52:01.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74cc24700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7480b0260 con 0x7fa7480afb30 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa746d9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa74806b530 0x7fa74806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:57180/0 (socket says 172.21.15.113:57180) 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa746d9d700 1 -- 172.21.15.113:0/2566544118 learned_addr learned my addr 172.21.15.113:0/2566544118 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa745d9b700 1 --2- 172.21.15.113:0/2566544118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7480afb30 0x7fa74806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74659c700 1 --2- 172.21.15.113:0/2566544118 >> v2:172.21.15.124:3300/0 conn(0x7fa74815e110 0x7fa74806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74559a700 1 -- 172.21.15.113:0/2566544118 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361303700 0 0) 0x7fa74806d980 con 0x7fa74806b530 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa74559a700 1 -- 172.21.15.113:0/2566544118 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7300035b0 con 0x7fa74806b530 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa745d9b700 1 -- 172.21.15.113:0/2566544118 >> v2:172.21.15.124:3300/0 conn(0x7fa74815e110 msgr2=0x7fa74806f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa745d9b700 1 --2- 172.21.15.113:0/2566544118 >> v2:172.21.15.124:3300/0 conn(0x7fa74815e110 0x7fa74806f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa745d9b700 1 -- 172.21.15.113:0/2566544118 >> v1:172.21.15.113:6789/0 conn(0x7fa74806b530 legacy=0x7fa74806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa745d9b700 1 -- 172.21.15.113:0/2566544118 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480b0540 con 0x7fa7480afb30 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.394+0000 7fa745d9b700 1 --2- 172.21.15.113:0/2566544118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7480afb30 0x7fa74806aff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009840 tx=0x7fa73c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97a48035032a53d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74559a700 1 -- 172.21.15.113:0/2566544118 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c011830 con 0x7fa7480afb30 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74559a700 1 -- 172.21.15.113:0/2566544118 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c011990 con 0x7fa7480afb30 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74559a700 1 -- 172.21.15.113:0/2566544118 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c011d00 con 0x7fa7480afb30 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 -- 172.21.15.113:0/2566544118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7480afb30 msgr2=0x7fa74806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 --2- 172.21.15.113:0/2566544118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7480afb30 0x7fa74806aff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009840 tx=0x7fa73c009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:01.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 -- 172.21.15.113:0/2566544118 shutdown_connections 2024-01-16T13:52:01.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 --2- 172.21.15.113:0/2566544118 >> v2:172.21.15.124:3300/0 conn(0x7fa74815e110 0x7fa74806f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 --2- 172.21.15.113:0/2566544118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7480afb30 0x7fa74806aff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 -- 172.21.15.113:0/2566544118 >> 172.21.15.113:0/2566544118 conn(0x7fa7480b3250 msgr2=0x7fa7480d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:01.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 -- 172.21.15.113:0/2566544118 shutdown_connections 2024-01-16T13:52:01.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.395+0000 7fa74cc24700 1 -- 172.21.15.113:0/2566544118 wait complete. 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 Processor -- start 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 -- start start 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa74806b530 0x7fa7481f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 0x7fa7480c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa74815e110 0x7fa7480ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74814bc80 con 0x7fa7480afb30 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74814bf60 con 0x7fa74815e110 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74cc24700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74814c240 con 0x7fa74806b530 2024-01-16T13:52:01.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa745d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 0x7fa7480c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa74659c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa74806b530 0x7fa7481f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa745d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 0x7fa7480c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33930/0 (socket says 172.21.15.113:33930) 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa745d9b700 1 -- 172.21.15.113:0/529803618 learned_addr learned my addr 172.21.15.113:0/529803618 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.397+0000 7fa746d9d700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa74815e110 0x7fa7480ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74659c700 1 -- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa74815e110 msgr2=0x7fa7480ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74659c700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa74815e110 0x7fa7480ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74659c700 1 -- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 msgr2=0x7fa7480c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74659c700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 0x7fa7480c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74659c700 1 -- 172.21.15.113:0/529803618 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c0094a0 con 0x7fa74806b530 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa745d9b700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 0x7fa7480c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74659c700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa74806b530 0x7fa7481f6180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa73800e870 tx=0x7fa73800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738013680 con 0x7fa74806b530 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7380137e0 con 0x7fa74806b530 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74814c520 con 0x7fa74806b530 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738013b90 con 0x7fa74806b530 2024-01-16T13:52:01.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.398+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7481f66c0 con 0x7fa74806b530 2024-01-16T13:52:01.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.399+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa748069020 con 0x7fa74806b530 2024-01-16T13:52:01.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.400+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fa738013cf0 con 0x7fa74806b530 2024-01-16T13:52:01.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.400+0000 7fa736ffd700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa72806b2c0 0x7fa72806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.400+0000 7fa745d9b700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa72806b2c0 0x7fa72806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.400+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa7380895f0 con 0x7fa74806b530 2024-01-16T13:52:01.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.400+0000 7fa745d9b700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa72806b2c0 0x7fa72806d780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009b90 tx=0x7fa73c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.408+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7380547f0 con 0x7fa74806b530 2024-01-16T13:52:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.603+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7480b3d60 con 0x7fa74806b530 2024-01-16T13:52:01.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.603+0000 7fa736ffd700 1 -- 172.21.15.113:0/529803618 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa738021020 con 0x7fa74806b530 2024-01-16T13:52:01.586 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa72806b2c0 msgr2=0x7fa72806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa72806b2c0 0x7fa72806d780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009b90 tx=0x7fa73c008030 comp rx=0 tx=0).stop 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa74806b530 msgr2=0x7fa7481f6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa74806b530 0x7fa7481f6180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa73800e870 tx=0x7fa73800eb80 comp rx=0 tx=0).stop 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 shutdown_connections 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa72806b2c0 0x7fa72806d780 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa74815e110 0x7fa7480ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa74806b530 0x7fa7481f6180 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 --2- 172.21.15.113:0/529803618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7480afb30 0x7fa7480c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.605+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 >> 172.21.15.113:0/529803618 conn(0x7fa7480b3250 msgr2=0x7fa748073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:01.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.606+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 shutdown_connections 2024-01-16T13:52:01.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.606+0000 7fa74cc24700 1 -- 172.21.15.113:0/529803618 wait complete. 2024-01-16T13:52:01.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T13:52:01.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.737+0000 7ff30188e700 1 Processor -- start 2024-01-16T13:52:01.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.737+0000 7ff30188e700 1 -- start start 2024-01-16T13:52:01.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff30188e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff2fc0d0870 0x7ff2fc0d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff30188e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff2fc0b6ef0 0x7ff2fc0b7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff30188e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc0d20b0 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff30188e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc0d3150 con 0x7ff2fc0d0870 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff30188e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc0d1240 con 0x7ff2fc0b6ef0 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff2fc0d0870 0x7ff2fc0d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff2fc0d0870 0x7ff2fc0d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33934/0 (socket says 172.21.15.113:33934) 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 -- 172.21.15.113:0/4006788820 learned_addr learned my addr 172.21.15.113:0/4006788820 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 -- 172.21.15.113:0/4006788820 >> v2:172.21.15.124:3300/0 conn(0x7ff2fc0b6ef0 msgr2=0x7ff2fc0b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fb7fe700 1 --2- 172.21.15.113:0/4006788820 >> v2:172.21.15.124:3300/0 conn(0x7ff2fc0b6ef0 0x7ff2fc0b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 --2- 172.21.15.113:0/4006788820 >> v2:172.21.15.124:3300/0 conn(0x7ff2fc0b6ef0 0x7ff2fc0b7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 -- 172.21.15.113:0/4006788820 >> v1:172.21.15.124:6789/0 conn(0x7ff2fc15e1e0 legacy=0x7ff2fc0d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.738+0000 7ff2fa7fc700 1 -- 172.21.15.113:0/4006788820 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc0d2390 con 0x7ff2fc0d0870 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff2fa7fc700 1 --2- 172.21.15.113:0/4006788820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc0d0870 0x7ff2fc0d0d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc08d120 tx=0x7ff2ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ae5c98dbcc7dfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff2f9ffb700 1 -- 172.21.15.113:0/4006788820 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec004230 con 0x7ff2fc0d0870 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff2f9ffb700 1 -- 172.21.15.113:0/4006788820 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec004390 con 0x7ff2fc0d0870 2024-01-16T13:52:01.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff2f9ffb700 1 -- 172.21.15.113:0/4006788820 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0036a0 con 0x7ff2fc0d0870 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 -- 172.21.15.113:0/4006788820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc0d0870 msgr2=0x7ff2fc0d0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 --2- 172.21.15.113:0/4006788820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc0d0870 0x7ff2fc0d0d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc08d120 tx=0x7ff2ec009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 -- 172.21.15.113:0/4006788820 shutdown_connections 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 --2- 172.21.15.113:0/4006788820 >> v2:172.21.15.124:3300/0 conn(0x7ff2fc0b6ef0 0x7ff2fc0b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 --2- 172.21.15.113:0/4006788820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc0d0870 0x7ff2fc0d0d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 -- 172.21.15.113:0/4006788820 >> 172.21.15.113:0/4006788820 conn(0x7ff2fc0b5b50 msgr2=0x7ff2fc0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 -- 172.21.15.113:0/4006788820 shutdown_connections 2024-01-16T13:52:01.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.739+0000 7ff30188e700 1 -- 172.21.15.113:0/4006788820 wait complete. 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.740+0000 7ff30188e700 1 Processor -- start 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 -- start start 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2fc0b6ef0 0x7ff2fc14e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff2fc0d0870 0x7ff2fc150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 0x7ff2fc0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc0cc020 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc14e8e0 con 0x7ff2fc0b6ef0 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff30188e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc0cb410 con 0x7ff2fc0d0870 2024-01-16T13:52:01.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.741+0000 7ff2fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 0x7ff2fc0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 0x7ff2fc0b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33948/0 (socket says 172.21.15.113:33948) 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 -- 172.21.15.113:0/603734345 learned_addr learned my addr 172.21.15.113:0/603734345 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fa7fc700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff2fc0d0870 0x7ff2fc150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2faffd700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2fc0b6ef0 0x7ff2fc14e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 -- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff2fc0d0870 msgr2=0x7ff2fc150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff2fc0d0870 0x7ff2fc150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 -- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2fc0b6ef0 msgr2=0x7ff2fc14e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2fc0b6ef0 0x7ff2fc14e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 -- 172.21.15.113:0/603734345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec009550 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2fb7fe700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 0x7ff2fc0b3d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400cd70 tx=0x7ff2f400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4003a20 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f4003b80 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc0cb6f0 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4004020 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.742+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc0cbcb0 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.743+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc0751e0 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.744+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7ff2f4004180 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.744+0000 7ff2e3fff700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff2e806b2c0 0x7ff2e806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:01.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.744+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff2f4012070 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.744+0000 7ff2faffd700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff2e806b2c0 0x7ff2e806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:01.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.744+0000 7ff2faffd700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff2e806b2c0 0x7ff2e806d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc161200 tx=0x7ff2f00090e0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:01.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.749+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2f4055110 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.945+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff2fc0b9510 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.945+0000 7ff2e3fff700 1 -- 172.21.15.113:0/603734345 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff2f4058730 con 0x7ff2fc15e1e0 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stdout: "poolnum": 1, 2024-01-16T13:52:01.928 INFO:teuthology.orchestra.run.smithi113.stdout: "poolname": ".mgr" 2024-01-16T13:52:01.929 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:01.929 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-16T13:52:01.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff2e806b2c0 msgr2=0x7ff2e806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff2e806b2c0 0x7ff2e806d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc161200 tx=0x7ff2f00090e0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 msgr2=0x7ff2fc0b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 0x7ff2fc0b3d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2f400cd70 tx=0x7ff2f400a570 comp rx=0 tx=0).stop 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 shutdown_connections 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff2e806b2c0 0x7ff2e806d780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc15e1e0 0x7ff2fc0b3d00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2fc0b6ef0 0x7ff2fc14e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 --2- 172.21.15.113:0/603734345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff2fc0d0870 0x7ff2fc150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 >> 172.21.15.113:0/603734345 conn(0x7ff2fc0b5b50 msgr2=0x7ff2fc0b83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 shutdown_connections 2024-01-16T13:52:01.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:01.947+0000 7ff30188e700 1 -- 172.21.15.113:0/603734345 wait complete. 2024-01-16T13:52:01.943 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T13:52:01.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T13:52:02.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.084+0000 7f661cd2a700 1 Processor -- start 2024-01-16T13:52:02.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.084+0000 7f661cd2a700 1 -- start start 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f661cd2a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6618073f90 0x7f661806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f661cd2a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f661806e9b0 0x7f661806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f661cd2a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661806ee40 con 0x7f66180b5850 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f661cd2a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66181470a0 con 0x7f661806e9b0 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f661cd2a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66180b3b50 con 0x7f6618073f90 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f6616d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f661806e9b0 0x7f661806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:02.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.085+0000 7f6616d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f661806e9b0 0x7f661806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33962/0 (socket says 172.21.15.113:33962) 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f6616d9d700 1 -- 172.21.15.113:0/183435223 learned_addr learned my addr 172.21.15.113:0/183435223 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f661659c700 1 --2- 172.21.15.113:0/183435223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6618073f90 0x7f661806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f6616d9d700 1 -- 172.21.15.113:0/183435223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6618073f90 msgr2=0x7f661806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f6616d9d700 1 --2- 172.21.15.113:0/183435223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6618073f90 0x7f661806f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f6616d9d700 1 -- 172.21.15.113:0/183435223 >> v1:172.21.15.124:6789/0 conn(0x7f66180b5850 legacy=0x7f6618074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f6616d9d700 1 -- 172.21.15.113:0/183435223 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66180b43b0 con 0x7f661806e9b0 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f6616d9d700 1 --2- 172.21.15.113:0/183435223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f661806e9b0 0x7f661806aff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6610009e50 tx=0x7f661000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3cc0f46c4fb066 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f661559a700 1 -- 172.21.15.113:0/183435223 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661000e070 con 0x7f661806e9b0 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f661559a700 1 -- 172.21.15.113:0/183435223 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661000b910 con 0x7f661806e9b0 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.086+0000 7f661559a700 1 -- 172.21.15.113:0/183435223 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661000bcc0 con 0x7f661806e9b0 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 -- 172.21.15.113:0/183435223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f661806e9b0 msgr2=0x7f661806aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 --2- 172.21.15.113:0/183435223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f661806e9b0 0x7f661806aff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6610009e50 tx=0x7f661000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 -- 172.21.15.113:0/183435223 shutdown_connections 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 --2- 172.21.15.113:0/183435223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f661806e9b0 0x7f661806aff0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 --2- 172.21.15.113:0/183435223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6618073f90 0x7f661806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 -- 172.21.15.113:0/183435223 >> 172.21.15.113:0/183435223 conn(0x7f66180d0b40 msgr2=0x7f66180d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:02.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 -- 172.21.15.113:0/183435223 shutdown_connections 2024-01-16T13:52:02.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.087+0000 7f661cd2a700 1 -- 172.21.15.113:0/183435223 wait complete. 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.088+0000 7f661cd2a700 1 Processor -- start 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.088+0000 7f661cd2a700 1 -- start start 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661cd2a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f661806e9b0 0x7f6618144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661cd2a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 0x7f661815f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661cd2a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f66180b5850 0x7f661815d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661cd2a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661815fd90 con 0x7f6618073f90 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661cd2a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618160070 con 0x7f661806e9b0 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661cd2a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6618160350 con 0x7f66180b5850 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f6615d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 0x7f661815f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f6615d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 0x7f661815f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33966/0 (socket says 172.21.15.113:33966) 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f6615d9b700 1 -- 172.21.15.113:0/2144919903 learned_addr learned my addr 172.21.15.113:0/2144919903 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:02.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f661659c700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f661806e9b0 0x7f6618144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.089+0000 7f6616d9d700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f66180b5850 0x7f661815d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6615d9b700 1 -- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f66180b5850 msgr2=0x7f661815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6615d9b700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f66180b5850 0x7f661815d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6615d9b700 1 -- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f661806e9b0 msgr2=0x7f6618144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6615d9b700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f661806e9b0 0x7f6618144c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6615d9b700 1 -- 172.21.15.113:0/2144919903 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610009b30 con 0x7f6618073f90 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6615d9b700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 0x7f661815f690 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f660800c450 tx=0x7f660800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66080052d0 con 0x7f6618073f90 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6608005430 con 0x7f6618073f90 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6618160630 con 0x7f6618073f90 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66080038c0 con 0x7f6618073f90 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.090+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661815dbc0 con 0x7f6618073f90 2024-01-16T13:52:02.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.091+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6618160b50 con 0x7f6618073f90 2024-01-16T13:52:02.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.091+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f6608003a20 con 0x7f6618073f90 2024-01-16T13:52:02.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.092+0000 7f6606ffd700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f65f806b2c0 0x7f65f806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:02.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.092+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6608088a10 con 0x7f6618073f90 2024-01-16T13:52:02.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.092+0000 7f661659c700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f65f806b2c0 0x7f65f806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:02.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.092+0000 7f661659c700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f65f806b2c0 0x7f65f806d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f660c009be0 tx=0x7f660c009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:02.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.096+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6618160b50 con 0x7f6618073f90 2024-01-16T13:52:02.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.285+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6618160b50 con 0x7f6618073f90 2024-01-16T13:52:02.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.881+0000 7f6606ffd700 1 -- 172.21.15.113:0/2144919903 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6618160b50 con 0x7f6618073f90 2024-01-16T13:52:02.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f65f806b2c0 msgr2=0x7f65f806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:02.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f65f806b2c0 0x7f65f806d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f660c009be0 tx=0x7f660c009210 comp rx=0 tx=0).stop 2024-01-16T13:52:02.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 msgr2=0x7f661815f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:02.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 0x7f661815f690 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f660800c450 tx=0x7f660800c760 comp rx=0 tx=0).stop 2024-01-16T13:52:02.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 shutdown_connections 2024-01-16T13:52:02.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f65f806b2c0 0x7f65f806d780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f66180b5850 0x7f661815d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f661806e9b0 0x7f6618144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 --2- 172.21.15.113:0/2144919903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6618073f90 0x7f661815f690 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 >> 172.21.15.113:0/2144919903 conn(0x7f66180d0b40 msgr2=0x7f66180b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 shutdown_connections 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:02.884+0000 7f661cd2a700 1 -- 172.21.15.113:0/2144919903 wait complete. 2024-01-16T13:52:02.868 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_metadata' created 2024-01-16T13:52:02.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T13:52:03.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.034+0000 7f82dd12d700 1 Processor -- start 2024-01-16T13:52:03.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.034+0000 7f82dd12d700 1 -- start start 2024-01-16T13:52:03.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82dd12d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d80b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82dd12d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f82d806aff0 0x7f82d8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82dd12d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d806c100 con 0x7f82d80744d0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82dd12d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d8074980 con 0x7f82d806aff0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82dd12d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80ae0f0 con 0x7f82d8145790 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d759e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f82d80744d0 0x7f82d815f8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:57188/0 (socket says 172.21.15.113:57188) 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d759e700 1 -- 172.21.15.113:0/3916281348 learned_addr learned my addr 172.21.15.113:0/3916281348 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d659c700 1 --2- 172.21.15.113:0/3916281348 >> v2:172.21.15.124:3300/0 conn(0x7f82d806aff0 0x7f82d8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d6d9d700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d80b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d5d9b700 1 -- 172.21.15.113:0/3916281348 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175296766 0 0) 0x7f82d806c100 con 0x7f82d80744d0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d5d9b700 1 -- 172.21.15.113:0/3916281348 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c00035b0 con 0x7f82d80744d0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d659c700 1 -- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 msgr2=0x7f82d80b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d659c700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d80b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d5d9b700 1 -- 172.21.15.113:0/3916281348 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187086997 0 0) 0x7f82c00035b0 con 0x7f82d80744d0 2024-01-16T13:52:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d659c700 1 -- 172.21.15.113:0/3916281348 >> v1:172.21.15.113:6789/0 conn(0x7f82d80744d0 legacy=0x7f82d815f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d659c700 1 -- 172.21.15.113:0/3916281348 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d80b0660 con 0x7f82d806aff0 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.035+0000 7f82d6d9d700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d80b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82d659c700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d806aff0 0x7f82d8073f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009870 tx=0x7f82c8009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4b1a54ec2f44e70 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82d5d9b700 1 -- 172.21.15.113:0/3916281348 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00019f0 con 0x7f82d806aff0 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82d5d9b700 1 -- 172.21.15.113:0/3916281348 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c8004030 con 0x7f82d806aff0 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82d5d9b700 1 -- 172.21.15.113:0/3916281348 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80043a0 con 0x7f82d806aff0 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 -- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d806aff0 msgr2=0x7f82d8073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d806aff0 0x7f82d8073f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009870 tx=0x7f82c8009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 -- 172.21.15.113:0/3916281348 shutdown_connections 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d80b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 --2- 172.21.15.113:0/3916281348 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d806aff0 0x7f82d8073f90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 -- 172.21.15.113:0/3916281348 >> 172.21.15.113:0/3916281348 conn(0x7f82d80b3250 msgr2=0x7f82d80d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 -- 172.21.15.113:0/3916281348 shutdown_connections 2024-01-16T13:52:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.036+0000 7f82dd12d700 1 -- 172.21.15.113:0/3916281348 wait complete. 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.038+0000 7f82dd12d700 1 Processor -- start 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.038+0000 7f82dd12d700 1 -- start start 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 0x7f82d813a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d80744d0 0x7f82d815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d806f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d806e050 con 0x7f82d806aff0 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d806e2a0 con 0x7f82d80744d0 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d806e580 con 0x7f82d8145790 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 0x7f82d813a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 0x7f82d813a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33980/0 (socket says 172.21.15.113:33980) 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 -- 172.21.15.113:0/1808882377 learned_addr learned my addr 172.21.15.113:0/1808882377 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:03.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d659c700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d80744d0 0x7f82d815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d759e700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d806f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 -- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 msgr2=0x7f82d806f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d806f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 -- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d80744d0 msgr2=0x7f82d815b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d80744d0 0x7f82d815b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 -- 172.21.15.113:0/1808882377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc0094a0 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82d6d9d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 0x7f82d813a4c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00ebe0 tx=0x7f82cc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc009760 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc00c850 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c80094a0 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00cbe0 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.039+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d806eb40 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.040+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d8069020 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.041+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f82cc00cd40 con 0x7f82d806aff0 2024-01-16T13:52:03.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.041+0000 7f82bf7fe700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f82c406b350 0x7f82c406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:03.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.041+0000 7f82d659c700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f82c406b350 0x7f82c406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:03.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.041+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f82cc089950 con 0x7f82d806aff0 2024-01-16T13:52:03.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.041+0000 7f82d659c700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f82c406b350 0x7f82c406d810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82c80097c0 tx=0x7f82c801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:03.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.046+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82cc054a30 con 0x7f82d806aff0 2024-01-16T13:52:03.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.235+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f82d815f8e0 con 0x7f82d806aff0 2024-01-16T13:52:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.884+0000 7f82bf7fe700 1 -- 172.21.15.113:0/1808882377 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f82cc058050 con 0x7f82d806aff0 2024-01-16T13:52:03.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f82c406b350 msgr2=0x7f82c406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:03.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f82c406b350 0x7f82c406d810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82c80097c0 tx=0x7f82c801c040 comp rx=0 tx=0).stop 2024-01-16T13:52:03.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 msgr2=0x7f82d813a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 0x7f82d813a4c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00ebe0 tx=0x7f82cc00eef0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 shutdown_connections 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f82c406b350 0x7f82c406d810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f82d8145790 0x7f82d806f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82d806aff0 0x7f82d813a4c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 --2- 172.21.15.113:0/1808882377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f82d80744d0 0x7f82d815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 >> 172.21.15.113:0/1808882377 conn(0x7f82d80b3250 msgr2=0x7f82d80b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 shutdown_connections 2024-01-16T13:52:03.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:03.887+0000 7f82dd12d700 1 -- 172.21.15.113:0/1808882377 wait complete. 2024-01-16T13:52:03.872 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_data' created 2024-01-16T13:52:03.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T13:52:04.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.043+0000 7f7bdee3e700 1 Processor -- start 2024-01-16T13:52:04.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.043+0000 7f7bdee3e700 1 -- start start 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdee3e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7bd8158600 0x7f7bd806f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdee3e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdee3e700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd806e4e0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdee3e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd806fac0 con 0x7f7bd8158600 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdee3e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd806e7c0 con 0x7f7bd815ee20 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7bd8158600 0x7f7bd806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bd7fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7bd80b4ef0 0x7f7bd815e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51906/0 (socket says 172.21.15.113:51906) 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7bd8158600 0x7f7bd806f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40228/0 (socket says 172.21.15.113:40228) 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bd7fff700 1 -- 172.21.15.113:0/3376898277 learned_addr learned my addr 172.21.15.113:0/3376898277 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdd3db700 1 --2- 172.21.15.113:0/3376898277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:04.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3376898277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 msgr2=0x7f7bd806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 --2- 172.21.15.113:0/3376898277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3376898277 >> v1:172.21.15.113:6789/0 conn(0x7f7bd80b4ef0 legacy=0x7f7bd815e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bd77fe700 1 -- 172.21.15.113:0/3376898277 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183117855 0 0) 0x7f7bd806e4e0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3376898277 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd806eaa0 con 0x7f7bd8158600 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.044+0000 7f7bdcbda700 1 --2- 172.21.15.113:0/3376898277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd8158600 0x7f7bd806f580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009870 tx=0x7f7bcc009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3693d15f6d5bdd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bd77fe700 1 -- 172.21.15.113:0/3376898277 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc011830 con 0x7f7bd8158600 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bd77fe700 1 -- 172.21.15.113:0/3376898277 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc011990 con 0x7f7bd8158600 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bd77fe700 1 -- 172.21.15.113:0/3376898277 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc011d20 con 0x7f7bd8158600 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3376898277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd8158600 msgr2=0x7f7bd806f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3376898277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd8158600 0x7f7bd806f580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009870 tx=0x7f7bcc009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3376898277 shutdown_connections 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3376898277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3376898277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd8158600 0x7f7bd806f580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3376898277 >> 172.21.15.113:0/3376898277 conn(0x7f7bd80b3250 msgr2=0x7f7bd80d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:04.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3376898277 shutdown_connections 2024-01-16T13:52:04.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.045+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3376898277 wait complete. 2024-01-16T13:52:04.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.046+0000 7f7bdee3e700 1 Processor -- start 2024-01-16T13:52:04.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.046+0000 7f7bdee3e700 1 -- start start 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdee3e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 0x7f7bd814d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdee3e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bd8158600 0x7f7bd8155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdee3e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd8143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdee3e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd8155fe0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdee3e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd814d950 con 0x7f7bd8158600 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdee3e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd814dc30 con 0x7f7bd815ee20 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdcbda700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 0x7f7bd814d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdcbda700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 0x7f7bd814d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40232/0 (socket says 172.21.15.113:40232) 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.047+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3414533951 learned_addr learned my addr 172.21.15.113:0/3414533951 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 msgr2=0x7f7bd8143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdd3db700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd8143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdcbda700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd8143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bd8158600 msgr2=0x7f7bd8155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdcbda700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bd8158600 0x7f7bd8155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdcbda700 1 -- 172.21.15.113:0/3414533951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0094a0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdcbda700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 0x7f7bd814d370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0090b0 tx=0x7f7bcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0036a0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc01f070 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc003a40 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd814df10 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.048+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd814e4d0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.049+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd8069020 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.049+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7bcc003ba0 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.050+0000 7f7bd57fa700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7bbc06b350 0x7f7bbc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.050+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7bcc021070 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.050+0000 7f7bd7fff700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7bbc06b350 0x7f7bbc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:04.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.050+0000 7f7bd7fff700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7bbc06b350 0x7f7bbc06d810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bc40011e0 tx=0x7f7bc40079b0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:04.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.054+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bcc056330 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.290+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7bd806e190 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.907+0000 7f7bd57fa700 1 -- 172.21.15.113:0/3414533951 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7bcc059950 con 0x7f7bd80b4ef0 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.911+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7bbc06b350 msgr2=0x7f7bbc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.911+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7bbc06b350 0x7f7bbc06d810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bc40011e0 tx=0x7f7bc40079b0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.911+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 msgr2=0x7f7bd814d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.911+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 0x7f7bd814d370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0090b0 tx=0x7f7bcc004750 comp rx=0 tx=0).stop 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.912+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 shutdown_connections 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.912+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7bbc06b350 0x7f7bbc06d810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.912+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7bd815ee20 0x7f7bd8143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.912+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7bd80b4ef0 0x7f7bd814d370 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0090b0 tx=0x7f7bcc004750 comp rx=0 tx=0).stop 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.912+0000 7f7bdee3e700 1 --2- 172.21.15.113:0/3414533951 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bd8158600 0x7f7bd8155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.912+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 >> 172.21.15.113:0/3414533951 conn(0x7f7bd80b3250 msgr2=0x7f7bd81562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:04.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.915+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 shutdown_connections 2024-01-16T13:52:04.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:04.916+0000 7f7bdee3e700 1 -- 172.21.15.113:0/3414533951 wait complete. 2024-01-16T13:52:04.901 INFO:teuthology.orchestra.run.smithi113.stderr:new fs with metadata pool 2 and data pool 3 2024-01-16T13:52:04.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:05.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.060+0000 7fb279de5700 1 Processor -- start 2024-01-16T13:52:05.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.060+0000 7fb279de5700 1 -- start start 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb279de5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb27415e080 0x7fb2740cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb279de5700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740d0150 con 0x7fb2740ce4d0 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb279de5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740cee50 con 0x7fb2740d22f0 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb279de5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27415e460 con 0x7fb27415e080 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb272ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb2740ce4d0 0x7fb2740d1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51912/0 (socket says 172.21.15.113:51912) 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb272ffd700 1 -- 172.21.15.113:0/2139157272 learned_addr learned my addr 172.21.15.113:0/2139157272 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2737fe700 1 --2- 172.21.15.113:0/2139157272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb27415e080 0x7fb2740cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454281748 0 0) 0x7fb2740d0150 con 0x7fb2740ce4d0 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2600035b0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587265311 0 0) 0x7fb2740cee50 con 0x7fb2740d22f0 2024-01-16T13:52:05.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2740d0150 con 0x7fb2740d22f0 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2737fe700 1 -- 172.21.15.113:0/2139157272 >> v1:172.21.15.124:6789/0 conn(0x7fb2740d22f0 legacy=0x7fb2740d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2737fe700 1 -- 172.21.15.113:0/2139157272 >> v1:172.21.15.113:6789/0 conn(0x7fb2740ce4d0 legacy=0x7fb2740d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1261361493 0 0) 0x7fb2600035b0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.061+0000 7fb2737fe700 1 -- 172.21.15.113:0/2139157272 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740cf130 con 0x7fb27415e080 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb2737fe700 1 --2- 172.21.15.113:0/2139157272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb27415e080 0x7fb2740cc2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009870 tx=0x7fb25c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7676dc261104d6be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264001670 con 0x7fb27415e080 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c004030 con 0x7fb27415e080 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb2727fc700 1 -- 172.21.15.113:0/2139157272 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0043a0 con 0x7fb27415e080 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 -- 172.21.15.113:0/2139157272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb27415e080 msgr2=0x7fb2740cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 --2- 172.21.15.113:0/2139157272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb27415e080 0x7fb2740cc2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009870 tx=0x7fb25c009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 -- 172.21.15.113:0/2139157272 shutdown_connections 2024-01-16T13:52:05.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 --2- 172.21.15.113:0/2139157272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb27415e080 0x7fb2740cc2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 -- 172.21.15.113:0/2139157272 >> 172.21.15.113:0/2139157272 conn(0x7fb274073130 msgr2=0x7fb27409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:05.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 -- 172.21.15.113:0/2139157272 shutdown_connections 2024-01-16T13:52:05.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.062+0000 7fb279de5700 1 -- 172.21.15.113:0/2139157272 wait complete. 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 Processor -- start 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 -- start start 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 0x7fb2740b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2740d22f0 0x7fb2740b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb27415e080 0x7fb27415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740b4910 con 0x7fb27415e080 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740b70a0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.064+0000 7fb279de5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740b7380 con 0x7fb2740d22f0 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb273fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb27415e080 0x7fb27415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 0x7fb2740b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb272ffd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2740d22f0 0x7fb2740b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 0x7fb2740b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57448/0 (socket says 172.21.15.113:57448) 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb273fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb27415e080 0x7fb27415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40236/0 (socket says 172.21.15.113:40236) 2024-01-16T13:52:05.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 -- 172.21.15.113:0/4288500853 learned_addr learned my addr 172.21.15.113:0/4288500853 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 -- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2740d22f0 msgr2=0x7fb2740b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2740d22f0 0x7fb2740b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 -- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb27415e080 msgr2=0x7fb27415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb27415e080 0x7fb27415b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 -- 172.21.15.113:0/4288500853 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0094a0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb2737fe700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 0x7fb2740b6a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb25c0090b0 tx=0x7fb25c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c003ad0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c01f070 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2740b4b80 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c003e90 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.065+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2740b55e0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.066+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27406b040 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.067+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb25c021030 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.067+0000 7fb26bfff700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb25406b350 0x7fb25406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.067+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb25c08adf0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.067+0000 7fb272ffd700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb25406b350 0x7fb25406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.067+0000 7fb272ffd700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb25406b350 0x7fb25406d810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb2741f7e80 tx=0x7fb264000f10 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.073+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb25c055d90 con 0x7fb2740ce4d0 2024-01-16T13:52:05.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.264+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb27409b290 con 0x7fb2740ce4d0 2024-01-16T13:52:05.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.266+0000 7fb26bfff700 1 -- 172.21.15.113:0/4288500853 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16554 (secure 0 0 0) 0x7fb25c0593b0 con 0x7fb2740ce4d0 2024-01-16T13:52:05.248 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:05.249 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:04.292131+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:05.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb25406b350 msgr2=0x7fb25406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb25406b350 0x7fb25406d810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb2741f7e80 tx=0x7fb264000f10 comp rx=0 tx=0).stop 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 msgr2=0x7fb2740b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 0x7fb2740b6a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb25c0090b0 tx=0x7fb25c004750 comp rx=0 tx=0).stop 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 shutdown_connections 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb25406b350 0x7fb25406d810 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb27415e080 0x7fb27415b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb2740ce4d0 0x7fb2740b6a10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 --2- 172.21.15.113:0/4288500853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2740d22f0 0x7fb2740b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 >> 172.21.15.113:0/4288500853 conn(0x7fb274073130 msgr2=0x7fb2740ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 shutdown_connections 2024-01-16T13:52:05.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.268+0000 7fb279de5700 1 -- 172.21.15.113:0/4288500853 wait complete. 2024-01-16T13:52:05.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:05.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.401+0000 7fec080f3700 1 Processor -- start 2024-01-16T13:52:05.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.401+0000 7fec080f3700 1 -- start start 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec080f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec001446a0 0x7fec000b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec080f3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec0006c710 0x7fec000b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec080f3700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec000af950 con 0x7fec000adf00 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec080f3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec00144a80 con 0x7fec0006c710 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec080f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec000afc30 con 0x7fec001446a0 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec0006c710 0x7fec000b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fec0006c710 0x7fec000b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40242/0 (socket says 172.21.15.113:40242) 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 -- 172.21.15.113:0/1689565730 learned_addr learned my addr 172.21.15.113:0/1689565730 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec05e8f700 1 --2- 172.21.15.113:0/1689565730 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec001446a0 0x7fec000b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 -- 172.21.15.113:0/1689565730 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec001446a0 msgr2=0x7fec000b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 --2- 172.21.15.113:0/1689565730 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec001446a0 0x7fec000b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 -- 172.21.15.113:0/1689565730 >> v1:172.21.15.124:6789/0 conn(0x7fec000adf00 legacy=0x7fec0006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 -- 172.21.15.113:0/1689565730 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec000aff10 con 0x7fec0006c710 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec06690700 1 --2- 172.21.15.113:0/1689565730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec0006c710 0x7fec000b3d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7febf800d150 tx=0x7febf800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7851f40a3793ae33 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.402+0000 7fec04e8d700 1 -- 172.21.15.113:0/1689565730 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8010070 con 0x7fec0006c710 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec04e8d700 1 -- 172.21.15.113:0/1689565730 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf800dd10 con 0x7fec0006c710 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec04e8d700 1 -- 172.21.15.113:0/1689565730 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf800b310 con 0x7fec0006c710 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 -- 172.21.15.113:0/1689565730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec0006c710 msgr2=0x7fec000b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 --2- 172.21.15.113:0/1689565730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec0006c710 0x7fec000b3d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7febf800d150 tx=0x7febf800d460 comp rx=0 tx=0).stop 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 -- 172.21.15.113:0/1689565730 shutdown_connections 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 --2- 172.21.15.113:0/1689565730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec0006c710 0x7fec000b3d40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 --2- 172.21.15.113:0/1689565730 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec001446a0 0x7fec000b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 -- 172.21.15.113:0/1689565730 >> 172.21.15.113:0/1689565730 conn(0x7fec00071b90 msgr2=0x7fec00071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 -- 172.21.15.113:0/1689565730 shutdown_connections 2024-01-16T13:52:05.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.403+0000 7fec080f3700 1 -- 172.21.15.113:0/1689565730 wait complete. 2024-01-16T13:52:05.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.404+0000 7fec080f3700 1 Processor -- start 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.404+0000 7fec080f3700 1 -- start start 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec080f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec0006c710 0x7fec001566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec080f3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec000adf00 0x7fec001581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec080f3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 0x7fec00158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec080f3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec001429b0 con 0x7fec001446a0 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec080f3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec000cd600 con 0x7fec000adf00 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec080f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec00141d30 con 0x7fec0006c710 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 0x7fec00158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 0x7fec00158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40246/0 (socket says 172.21.15.113:40246) 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 -- 172.21.15.113:0/1019363199 learned_addr learned my addr 172.21.15.113:0/1019363199 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec05e8f700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec0006c710 0x7fec001566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec0568e700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec000adf00 0x7fec001581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 -- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec0006c710 msgr2=0x7fec001566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec0006c710 0x7fec001566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 -- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec000adf00 msgr2=0x7fec001581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec000adf00 0x7fec001581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec06690700 1 -- 172.21.15.113:0/1019363199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf800bd40 con 0x7fec001446a0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.405+0000 7fec0568e700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec000adf00 0x7fec001581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.406+0000 7fec06690700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 0x7fec00158730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fec000729d0 tx=0x7febf8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.406+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf800bbc0 con 0x7fec001446a0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.406+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf800b4c0 con 0x7fec001446a0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.406+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec00142010 con 0x7fec001446a0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.406+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8004040 con 0x7fec001446a0 2024-01-16T13:52:05.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.406+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec001425d0 con 0x7fec001446a0 2024-01-16T13:52:05.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.407+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec00142790 con 0x7fec001446a0 2024-01-16T13:52:05.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.407+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7febf80041a0 con 0x7fec001446a0 2024-01-16T13:52:05.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.407+0000 7febf67fc700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7febe406b350 0x7febe406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.408+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febf808d640 con 0x7fec001446a0 2024-01-16T13:52:05.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.408+0000 7fec05e8f700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7febe406b350 0x7febe406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.408+0000 7fec05e8f700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7febe406b350 0x7febe406d810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7febfc009b40 tx=0x7febfc009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.412+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febf8058540 con 0x7fec001446a0 2024-01-16T13:52:05.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.603+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec000753f0 con 0x7fec001446a0 2024-01-16T13:52:05.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.604+0000 7febf67fc700 1 -- 172.21.15.113:0/1019363199 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16554 (secure 0 0 0) 0x7febf805bb60 con 0x7fec001446a0 2024-01-16T13:52:05.586 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:05.586 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:04.292131+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7febe406b350 msgr2=0x7febe406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7febe406b350 0x7febe406d810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7febfc009b40 tx=0x7febfc009210 comp rx=0 tx=0).stop 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 msgr2=0x7fec00158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 0x7fec00158730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fec000729d0 tx=0x7febf8004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 shutdown_connections 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7febe406b350 0x7febe406d810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec001446a0 0x7fec00158730 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec0006c710 0x7fec001566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 --2- 172.21.15.113:0/1019363199 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec000adf00 0x7fec001581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 >> 172.21.15.113:0/1019363199 conn(0x7fec00071b90 msgr2=0x7fec000b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 shutdown_connections 2024-01-16T13:52:05.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.606+0000 7fec080f3700 1 -- 172.21.15.113:0/1019363199 wait complete. 2024-01-16T13:52:05.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T13:52:05.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.743+0000 7f1345ba0700 1 Processor -- start 2024-01-16T13:52:05.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.744+0000 7f1345ba0700 1 -- start start 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.744+0000 7f1345ba0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f134015ed90 0x7f134006fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.744+0000 7f1345ba0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1340070370 con 0x7f13400b5540 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.744+0000 7f1345ba0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134006e8c0 con 0x7f13400b5160 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.744+0000 7f1345ba0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134006eba0 con 0x7f134015ed90 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f134539f700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f13400b5540 0x7f13400cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51928/0 (socket says 172.21.15.113:51928) 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f134539f700 1 -- 172.21.15.113:0/9534095 learned_addr learned my addr 172.21.15.113:0/9534095 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133ffff700 1 --2- 172.21.15.113:0/9534095 >> v2:172.21.15.124:3300/0 conn(0x7f134015ed90 0x7f134006fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133f7fe700 1 -- 172.21.15.113:0/9534095 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851439895 0 0) 0x7f1340070370 con 0x7f13400b5540 2024-01-16T13:52:05.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133f7fe700 1 -- 172.21.15.113:0/9534095 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13300035b0 con 0x7f13400b5540 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133ffff700 1 -- 172.21.15.113:0/9534095 >> v1:172.21.15.124:6789/0 conn(0x7f13400b5160 legacy=0x7f134015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133ffff700 1 -- 172.21.15.113:0/9534095 >> v1:172.21.15.113:6789/0 conn(0x7f13400b5540 legacy=0x7f13400cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133ffff700 1 -- 172.21.15.113:0/9534095 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134006ee80 con 0x7f134015ed90 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133ffff700 1 --2- 172.21.15.113:0/9534095 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f134015ed90 0x7f134006fd70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f132c009840 tx=0x7f132c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a65a22c235f09b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133f7fe700 1 -- 172.21.15.113:0/9534095 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c011830 con 0x7f134015ed90 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133f7fe700 1 -- 172.21.15.113:0/9534095 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c011990 con 0x7f134015ed90 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.745+0000 7f133f7fe700 1 -- 172.21.15.113:0/9534095 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c011d00 con 0x7f134015ed90 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 -- 172.21.15.113:0/9534095 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f134015ed90 msgr2=0x7f134006fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 --2- 172.21.15.113:0/9534095 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f134015ed90 0x7f134006fd70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f132c009840 tx=0x7f132c009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 -- 172.21.15.113:0/9534095 shutdown_connections 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 --2- 172.21.15.113:0/9534095 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f134015ed90 0x7f134006fd70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 -- 172.21.15.113:0/9534095 >> 172.21.15.113:0/9534095 conn(0x7f13400727c0 msgr2=0x7f134006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 -- 172.21.15.113:0/9534095 shutdown_connections 2024-01-16T13:52:05.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.746+0000 7f1345ba0700 1 -- 172.21.15.113:0/9534095 wait complete. 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.747+0000 7f1345ba0700 1 Processor -- start 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.747+0000 7f1345ba0700 1 -- start start 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1345ba0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f13400b5160 0x7f13401f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1345ba0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f13400b5540 0x7f134014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1345ba0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 0x7f134014a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1345ba0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134015d0f0 con 0x7f134015ed90 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1345ba0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134015c790 con 0x7f13400b5160 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1345ba0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134014e780 con 0x7f13400b5540 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 0x7f134014a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 0x7f134014a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40262/0 (socket says 172.21.15.113:40262) 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 -- 172.21.15.113:0/2461631408 learned_addr learned my addr 172.21.15.113:0/2461631408 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f1344b9e700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f13400b5160 0x7f13401f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f133ffff700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f13400b5540 0x7f134014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 -- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f13400b5540 msgr2=0x7f134014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f13400b5540 0x7f134014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 -- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f13400b5160 msgr2=0x7f13401f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f13400b5160 0x7f13401f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.748+0000 7f134539f700 1 -- 172.21.15.113:0/2461631408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0094a0 con 0x7f134015ed90 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.749+0000 7f134539f700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 0x7f134014a510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f133800cbb0 tx=0x7f133800cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.749+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133800a9e0 con 0x7f134015ed90 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.749+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133800ab40 con 0x7f134015ed90 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.749+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134015ca30 con 0x7f134015ed90 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.749+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338012400 con 0x7f134015ed90 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.749+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134015d340 con 0x7f134015ed90 2024-01-16T13:52:05.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.750+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1340068e70 con 0x7f134015ed90 2024-01-16T13:52:05.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.750+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f133801b3b0 con 0x7f134015ed90 2024-01-16T13:52:05.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.750+0000 7f133d7fa700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f132406b350 0x7f132406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:05.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.751+0000 7f1344b9e700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f132406b350 0x7f132406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:05.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.751+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1338089780 con 0x7f134015ed90 2024-01-16T13:52:05.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.751+0000 7f1344b9e700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f132406b350 0x7f132406d810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1334001800 tx=0x7f1334009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:05.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.755+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1338054690 con 0x7f134015ed90 2024-01-16T13:52:05.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:05.993+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f134015d530 con 0x7f134015ed90 2024-01-16T13:52:06.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.922+0000 7f133d7fa700 1 -- 172.21.15.113:0/2461631408 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1338057cb0 con 0x7f134015ed90 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f132406b350 msgr2=0x7f132406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f132406b350 0x7f132406d810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1334001800 tx=0x7f1334009040 comp rx=0 tx=0).stop 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 msgr2=0x7f134014a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 0x7f134014a510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f133800cbb0 tx=0x7f133800cec0 comp rx=0 tx=0).stop 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 shutdown_connections 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f132406b350 0x7f132406d810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f134015ed90 0x7f134014a510 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f13400b5160 0x7f13401f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 --2- 172.21.15.113:0/2461631408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f13400b5540 0x7f134014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.925+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 >> 172.21.15.113:0/2461631408 conn(0x7f13400727c0 msgr2=0x7f13400b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.926+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 shutdown_connections 2024-01-16T13:52:06.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:06.926+0000 7f1345ba0700 1 -- 172.21.15.113:0/2461631408 wait complete. 2024-01-16T13:52:06.922 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T13:52:06.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T13:52:07.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 Processor -- start 2024-01-16T13:52:07.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 -- start start 2024-01-16T13:52:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f4415e0c0 0x7f5f440afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5f440746a0 0x7f5f440add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4415e4a0 con 0x7f5f440b3bf0 2024-01-16T13:52:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4406aff0 con 0x7f5f440746a0 2024-01-16T13:52:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.071+0000 7f5f4aac9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4406b2d0 con 0x7f5f4415e0c0 2024-01-16T13:52:07.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f43fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f5f440b3bf0 0x7f5f44073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51944/0 (socket says 172.21.15.113:51944) 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f43fff700 1 -- 172.21.15.113:0/227803207 learned_addr learned my addr 172.21.15.113:0/227803207 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f49066700 1 --2- 172.21.15.113:0/227803207 >> v2:172.21.15.124:3300/0 conn(0x7f5f440746a0 0x7f5f440add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f48865700 1 --2- 172.21.15.113:0/227803207 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f4415e0c0 0x7f5f440afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f437fe700 1 -- 172.21.15.113:0/227803207 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259052623 0 0) 0x7f5f4415e4a0 con 0x7f5f440b3bf0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f437fe700 1 -- 172.21.15.113:0/227803207 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f340035b0 con 0x7f5f440b3bf0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f48865700 1 -- 172.21.15.113:0/227803207 >> v2:172.21.15.124:3300/0 conn(0x7f5f440746a0 msgr2=0x7f5f440add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f48865700 1 --2- 172.21.15.113:0/227803207 >> v2:172.21.15.124:3300/0 conn(0x7f5f440746a0 0x7f5f440add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f437fe700 1 -- 172.21.15.113:0/227803207 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 182780023 0 0) 0x7f5f340035b0 con 0x7f5f440b3bf0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f48865700 1 -- 172.21.15.113:0/227803207 >> v1:172.21.15.113:6789/0 conn(0x7f5f440b3bf0 legacy=0x7f5f44073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f48865700 1 -- 172.21.15.113:0/227803207 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4406b5b0 con 0x7f5f4415e0c0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f49066700 1 --2- 172.21.15.113:0/227803207 >> v2:172.21.15.124:3300/0 conn(0x7f5f440746a0 0x7f5f440add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f48865700 1 --2- 172.21.15.113:0/227803207 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f4415e0c0 0x7f5f440afb80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009840 tx=0x7f5f38009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee9040b446d2d885 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:07.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f437fe700 1 -- 172.21.15.113:0/227803207 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30001670 con 0x7f5f4415e0c0 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f437fe700 1 -- 172.21.15.113:0/227803207 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38004030 con 0x7f5f4415e0c0 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.072+0000 7f5f437fe700 1 -- 172.21.15.113:0/227803207 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f380043a0 con 0x7f5f4415e0c0 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 -- 172.21.15.113:0/227803207 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f4415e0c0 msgr2=0x7f5f440afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/227803207 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f4415e0c0 0x7f5f440afb80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009840 tx=0x7f5f38009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 -- 172.21.15.113:0/227803207 shutdown_connections 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/227803207 >> v2:172.21.15.124:3300/0 conn(0x7f5f440746a0 0x7f5f440add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/227803207 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f4415e0c0 0x7f5f440afb80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 -- 172.21.15.113:0/227803207 >> 172.21.15.113:0/227803207 conn(0x7f5f440d0c00 msgr2=0x7f5f440d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 -- 172.21.15.113:0/227803207 shutdown_connections 2024-01-16T13:52:07.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.073+0000 7f5f4aac9700 1 -- 172.21.15.113:0/227803207 wait complete. 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.074+0000 7f5f4aac9700 1 Processor -- start 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.074+0000 7f5f4aac9700 1 -- start start 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f4aac9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 0x7f5f440cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f4aac9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f440b3bf0 0x7f5f4415c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f4aac9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f4415e0c0 0x7f5f44147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f4aac9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f441480f0 con 0x7f5f440746a0 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f4aac9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f440cdfa0 con 0x7f5f4415e0c0 2024-01-16T13:52:07.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f4aac9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f440ce280 con 0x7f5f440b3bf0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f48865700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 0x7f5f440cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f48865700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 0x7f5f440cfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40276/0 (socket says 172.21.15.113:40276) 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f48865700 1 -- 172.21.15.113:0/238555812 learned_addr learned my addr 172.21.15.113:0/238555812 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f49066700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f4415e0c0 0x7f5f44147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.075+0000 7f5f43fff700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f440b3bf0 0x7f5f4415c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f48865700 1 -- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f440b3bf0 msgr2=0x7f5f4415c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f48865700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f440b3bf0 0x7f5f4415c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f48865700 1 -- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f4415e0c0 msgr2=0x7f5f44147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f48865700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f4415e0c0 0x7f5f44147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f48865700 1 -- 172.21.15.113:0/238555812 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c009a70 con 0x7f5f440746a0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f49066700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f4415e0c0 0x7f5f44147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f48865700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 0x7f5f440cfba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5f380090b0 tx=0x7f5f38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38003bf0 con 0x7f5f440746a0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3801f070 con 0x7f5f440746a0 2024-01-16T13:52:07.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f380094a0 con 0x7f5f440746a0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38011630 con 0x7f5f440746a0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.076+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f440ce840 con 0x7f5f440746a0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.077+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f440b4010 con 0x7f5f440746a0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.077+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5f38021030 con 0x7f5f440746a0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.078+0000 7f5f417fa700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f2806b350 0x7f5f2806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.078+0000 7f5f43fff700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f2806b350 0x7f5f2806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.078+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f3808af50 con 0x7f5f440746a0 2024-01-16T13:52:07.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.078+0000 7f5f43fff700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f2806b350 0x7f5f2806d810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f30001eb0 tx=0x7f5f30001c60 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:07.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.082+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f38055e50 con 0x7f5f440746a0 2024-01-16T13:52:07.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.315+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5f44069020 con 0x7f5f440746a0 2024-01-16T13:52:07.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.928+0000 7f5f417fa700 1 -- 172.21.15.113:0/238555812 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5f38059470 con 0x7f5f440746a0 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f2806b350 msgr2=0x7f5f2806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f2806b350 0x7f5f2806d810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f30001eb0 tx=0x7f5f30001c60 comp rx=0 tx=0).stop 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 msgr2=0x7f5f440cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 0x7f5f440cfba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5f380090b0 tx=0x7f5f38004750 comp rx=0 tx=0).stop 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 shutdown_connections 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f2806b350 0x7f5f2806d810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f4415e0c0 0x7f5f44147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f440746a0 0x7f5f440cfba0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 --2- 172.21.15.113:0/238555812 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f440b3bf0 0x7f5f4415c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:07.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 >> 172.21.15.113:0/238555812 conn(0x7f5f440d0c00 msgr2=0x7f5f4406f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:07.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 shutdown_connections 2024-01-16T13:52:07.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:07.931+0000 7f5f4aac9700 1 -- 172.21.15.113:0/238555812 wait complete. 2024-01-16T13:52:07.927 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T13:52:07.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:08.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.069+0000 7fb6769b2700 1 Processor -- start 2024-01-16T13:52:08.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.069+0000 7fb6769b2700 1 -- start start 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb6769b2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015be00 0x7fb6700cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb6769b2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb6700d0040 0x7fb6700d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb6769b2700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6700d09d0 con 0x7fb67015c1e0 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb6769b2700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6700b00e0 con 0x7fb6700d0040 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb6769b2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6700af200 con 0x7fb67015be00 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb674f4f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb6700d0040 0x7fb6700d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb674f4f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb6700d0040 0x7fb6700d0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40282/0 (socket says 172.21.15.113:40282) 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb674f4f700 1 -- 172.21.15.113:0/901608151 learned_addr learned my addr 172.21.15.113:0/901608151 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:08.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.070+0000 7fb66ffff700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015be00 0x7fb6700cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb674f4f700 1 -- 172.21.15.113:0/901608151 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015be00 msgr2=0x7fb6700cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb674f4f700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015be00 0x7fb6700cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb674f4f700 1 -- 172.21.15.113:0/901608151 >> v1:172.21.15.124:6789/0 conn(0x7fb67015c1e0 legacy=0x7fb6700aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb674f4f700 1 -- 172.21.15.113:0/901608151 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700af4d0 con 0x7fb6700d0040 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb66ffff700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015be00 0x7fb6700cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb674f4f700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6700d0490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb668009e50 tx=0x7fb66800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31633df80fa182d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb66effd700 1 -- 172.21.15.113:0/901608151 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66800e070 con 0x7fb6700d0040 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb66effd700 1 -- 172.21.15.113:0/901608151 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66800b9c0 con 0x7fb6700d0040 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb66effd700 1 -- 172.21.15.113:0/901608151 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66800bd70 con 0x7fb6700d0040 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 -- 172.21.15.113:0/901608151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 msgr2=0x7fb6700d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6700d0490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb668009e50 tx=0x7fb66800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 -- 172.21.15.113:0/901608151 shutdown_connections 2024-01-16T13:52:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6700d0490 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 --2- 172.21.15.113:0/901608151 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015be00 0x7fb6700cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 -- 172.21.15.113:0/901608151 >> 172.21.15.113:0/901608151 conn(0x7fb67009aa00 msgr2=0x7fb67009ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 -- 172.21.15.113:0/901608151 shutdown_connections 2024-01-16T13:52:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.071+0000 7fb6769b2700 1 -- 172.21.15.113:0/901608151 wait complete. 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.073+0000 7fb6769b2700 1 Processor -- start 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.073+0000 7fb6769b2700 1 -- start start 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb6769b2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6701f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb6769b2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb67015be00 0x7fb6701580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb6769b2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015c1e0 0x7fb6701f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb6769b2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701f4ea0 con 0x7fb6700d0040 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb6769b2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701f5460 con 0x7fb67015be00 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb6769b2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6701f2320 con 0x7fb67015c1e0 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6701f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6701f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40292/0 (socket says 172.21.15.113:40292) 2024-01-16T13:52:08.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 -- 172.21.15.113:0/2449819106 learned_addr learned my addr 172.21.15.113:0/2449819106 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66f7fe700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb67015be00 0x7fb6701580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb674f4f700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015c1e0 0x7fb6701f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 -- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015c1e0 msgr2=0x7fb6701f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015c1e0 0x7fb6701f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 -- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb67015be00 msgr2=0x7fb6701580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb67015be00 0x7fb6701580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.074+0000 7fb66ffff700 1 -- 172.21.15.113:0/2449819106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb668009b30 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.075+0000 7fb66ffff700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6701f58f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb66000e840 tx=0x7fb66000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.075+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660013610 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.075+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb660013770 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.075+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6701f2600 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.075+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660013b20 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.075+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb670158630 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.076+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6701f2840 con 0x7fb6700d0040 2024-01-16T13:52:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.076+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb660013c80 con 0x7fb6700d0040 2024-01-16T13:52:08.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.076+0000 7fb66cff9700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb65006b350 0x7fb65006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.077+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb660089610 con 0x7fb6700d0040 2024-01-16T13:52:08.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.077+0000 7fb66f7fe700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb65006b350 0x7fb65006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.077+0000 7fb66f7fe700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb65006b350 0x7fb65006d810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb6700c87b0 tx=0x7fb658009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.081+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6701f2840 con 0x7fb6700d0040 2024-01-16T13:52:08.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.317+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb670067f60 con 0x7fb6700d0040 2024-01-16T13:52:08.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.318+0000 7fb66cff9700 1 -- 172.21.15.113:0/2449819106 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fb660057fa0 con 0x7fb6700d0040 2024-01-16T13:52:08.300 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:08.300 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb65006b350 msgr2=0x7fb65006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb65006b350 0x7fb65006d810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb6700c87b0 tx=0x7fb658009040 comp rx=0 tx=0).stop 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 msgr2=0x7fb6701f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6701f58f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb66000e840 tx=0x7fb66000eb50 comp rx=0 tx=0).stop 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 shutdown_connections 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb65006b350 0x7fb65006d810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb67015c1e0 0x7fb6701f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6700d0040 0x7fb6701f58f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 --2- 172.21.15.113:0/2449819106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb67015be00 0x7fb6701580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 >> 172.21.15.113:0/2449819106 conn(0x7fb67009aa00 msgr2=0x7fb6700d1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:08.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 shutdown_connections 2024-01-16T13:52:08.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.320+0000 7fb6769b2700 1 -- 172.21.15.113:0/2449819106 wait complete. 2024-01-16T13:52:08.303 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:08.314 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:08.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.453+0000 7f32ec3cc700 1 Processor -- start 2024-01-16T13:52:08.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.453+0000 7f32ec3cc700 1 -- start start 2024-01-16T13:52:08.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32ec3cc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f32e40d2160 0x7f32e40b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32ec3cc700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e40cd100 con 0x7f32e41603c0 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32ec3cc700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e40d09a0 con 0x7f32e415bed0 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32ec3cc700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e415c2b0 con 0x7f32e40d2160 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32e9967700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f32e40d2160 0x7f32e40b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32ea969700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f32e41603c0 0x7f32e40cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51960/0 (socket says 172.21.15.113:51960) 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.454+0000 7f32ea969700 1 -- 172.21.15.113:0/1707430162 learned_addr learned my addr 172.21.15.113:0/1707430162 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9967700 1 -- 172.21.15.113:0/1707430162 >> v1:172.21.15.124:6789/0 conn(0x7f32e415bed0 legacy=0x7f32e40d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9967700 1 -- 172.21.15.113:0/1707430162 >> v1:172.21.15.113:6789/0 conn(0x7f32e41603c0 legacy=0x7f32e40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9967700 1 -- 172.21.15.113:0/1707430162 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4160850 con 0x7f32e40d2160 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9967700 1 --2- 172.21.15.113:0/1707430162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e40d2160 0x7f32e40b19f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009840 tx=0x7f32e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb46650f2813f8dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9166700 1 -- 172.21.15.113:0/1707430162 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0004230 con 0x7f32e40d2160 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9166700 1 -- 172.21.15.113:0/1707430162 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e0004390 con 0x7f32e40d2160 2024-01-16T13:52:08.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32e9166700 1 -- 172.21.15.113:0/1707430162 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e00036a0 con 0x7f32e40d2160 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 -- 172.21.15.113:0/1707430162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e40d2160 msgr2=0x7f32e40b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/1707430162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e40d2160 0x7f32e40b19f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009840 tx=0x7f32e0009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 -- 172.21.15.113:0/1707430162 shutdown_connections 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/1707430162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e40d2160 0x7f32e40b19f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 -- 172.21.15.113:0/1707430162 >> 172.21.15.113:0/1707430162 conn(0x7f32e4073130 msgr2=0x7f32e409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 -- 172.21.15.113:0/1707430162 shutdown_connections 2024-01-16T13:52:08.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.455+0000 7f32ec3cc700 1 -- 172.21.15.113:0/1707430162 wait complete. 2024-01-16T13:52:08.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 Processor -- start 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 -- start start 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f32e40d2160 0x7f32e4147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 0x7f32e414a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32e41603c0 0x7f32e406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e4147bd0 con 0x7f32e415bed0 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40b64f0 con 0x7f32e41603c0 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32ec3cc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40c7e20 con 0x7f32e40d2160 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32e9967700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 0x7f32e414a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.457+0000 7f32e9967700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 0x7f32e414a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40306/0 (socket says 172.21.15.113:40306) 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 -- 172.21.15.113:0/2517531886 learned_addr learned my addr 172.21.15.113:0/2517531886 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32ea168700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f32e40d2160 0x7f32e4147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32ea969700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32e41603c0 0x7f32e406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 -- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f32e40d2160 msgr2=0x7f32e4147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f32e40d2160 0x7f32e4147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 -- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32e41603c0 msgr2=0x7f32e406fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32e41603c0 0x7f32e406fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 -- 172.21.15.113:0/2517531886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00094a0 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32e9967700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 0x7f32e414a890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009b90 tx=0x7f32e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0003ea0 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e001e070 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e40b5ee0 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0011870 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.458+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e40c80c0 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.459+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e4067f60 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.459+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f32e00119d0 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.460+0000 7f32daffd700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f32c806b350 0x7f32c806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.460+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32e008ad30 con 0x7f32e415bed0 2024-01-16T13:52:08.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.460+0000 7f32ea168700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f32c806b350 0x7f32c806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.460+0000 7f32ea168700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f32c806b350 0x7f32c806d810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32d4007fd0 tx=0x7f32d4007ec0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.464+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32e0055c30 con 0x7f32e415bed0 2024-01-16T13:52:08.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.657+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32e40cc230 con 0x7f32e415bed0 2024-01-16T13:52:08.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.658+0000 7f32daffd700 1 -- 172.21.15.113:0/2517531886 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f32e0020020 con 0x7f32e415bed0 2024-01-16T13:52:08.640 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:08.641 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f32c806b350 msgr2=0x7f32c806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f32c806b350 0x7f32c806d810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32d4007fd0 tx=0x7f32d4007ec0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 msgr2=0x7f32e414a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 0x7f32e414a890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009b90 tx=0x7f32e0004750 comp rx=0 tx=0).stop 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 shutdown_connections 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f32c806b350 0x7f32c806d810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32e41603c0 0x7f32e406fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f32e40d2160 0x7f32e4147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 --2- 172.21.15.113:0/2517531886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32e415bed0 0x7f32e414a890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 >> 172.21.15.113:0/2517531886 conn(0x7f32e4073130 msgr2=0x7f32e40b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 shutdown_connections 2024-01-16T13:52:08.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.660+0000 7f32ec3cc700 1 -- 172.21.15.113:0/2517531886 wait complete. 2024-01-16T13:52:08.655 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T13:52:08.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:52:08.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.794+0000 7f7f437d5700 1 Processor -- start 2024-01-16T13:52:08.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.794+0000 7f7f437d5700 1 -- start start 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f437d5700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7f3c15e1e0 0x7f7f3c0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f437d5700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c0d20b0 con 0x7f7f3c0d0f80 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f437d5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c0d2390 con 0x7f7f3c0d3a00 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f437d5700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d3150 con 0x7f7f3c15e1e0 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f41d72700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7f3c0d0f80 0x7f7f3c0b6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51968/0 (socket says 172.21.15.113:51968) 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f41d72700 1 -- 172.21.15.113:0/1174462353 learned_addr learned my addr 172.21.15.113:0/1174462353 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f41571700 1 --2- 172.21.15.113:0/1174462353 >> v2:172.21.15.124:3300/0 conn(0x7f7f3c15e1e0 0x7f7f3c0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776218348 0 0) 0x7f7f3c0d20b0 con 0x7f7f3c0d0f80 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f280035b0 con 0x7f7f3c0d0f80 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2234472769 0 0) 0x7f7f280035b0 con 0x7f7f3c0d0f80 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 >> v2:172.21.15.124:3300/0 conn(0x7f7f3c15e1e0 msgr2=0x7f7f3c0d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 --2- 172.21.15.113:0/1174462353 >> v2:172.21.15.124:3300/0 conn(0x7f7f3c15e1e0 0x7f7f3c0d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 >> v1:172.21.15.124:6789/0 conn(0x7f7f3c0d3a00 legacy=0x7f7f3c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0d2670 con 0x7f7f3c0d0f80 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f41571700 1 --2- 172.21.15.113:0/1174462353 >> v2:172.21.15.124:3300/0 conn(0x7f7f3c15e1e0 0x7f7f3c0d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f7f340019f0 con 0x7f7f3c0d0f80 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 >> v1:172.21.15.113:6789/0 conn(0x7f7f3c0d0f80 legacy=0x7f7f3c0b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f28003a00 0x7f7f28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f28004850 0x7f7f28004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28005210 0x7f7f2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d3150 con 0x7f7f28003a00 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2800c0c0 con 0x7f7f28004850 2024-01-16T13:52:08.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2800c280 con 0x7f7f28005210 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.795+0000 7f7f41571700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f28003a00 0x7f7f28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f40d70700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f28004850 0x7f7f28004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41d72700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28005210 0x7f7f2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41571700 1 -- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28005210 msgr2=0x7f7f2800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41571700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28005210 0x7f7f2800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41571700 1 -- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f28004850 msgr2=0x7f7f28004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41571700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f28004850 0x7f7f28004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41571700 1 -- 172.21.15.113:0/1174462353 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280035b0 con 0x7f7f28003a00 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41d72700 1 -- 172.21.15.113:0/1174462353 reap_dead start 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f40d70700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f28004850 0x7f7f28004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f41571700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f28003a00 0x7f7f28003e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009d30 tx=0x7f7f28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6318a143bfa5d5e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38009e80 con 0x7f7f28003a00 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3800e9a0 con 0x7f7f28003a00 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f33fff700 1 -- 172.21.15.113:0/1174462353 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3800ed10 con 0x7f7f28003a00 2024-01-16T13:52:08.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f437d5700 1 -- 172.21.15.113:0/1174462353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f28003a00 msgr2=0x7f7f28003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.796+0000 7f7f437d5700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f28003a00 0x7f7f28003e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009d30 tx=0x7f7f28005880 comp rx=0 tx=0).stop 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.797+0000 7f7f437d5700 1 -- 172.21.15.113:0/1174462353 shutdown_connections 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.797+0000 7f7f437d5700 1 --2- 172.21.15.113:0/1174462353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f28003a00 0x7f7f28003e20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.797+0000 7f7f437d5700 1 -- 172.21.15.113:0/1174462353 >> 172.21.15.113:0/1174462353 conn(0x7f7f3c0b5b50 msgr2=0x7f7f3c0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.797+0000 7f7f437d5700 1 -- 172.21.15.113:0/1174462353 shutdown_connections 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.797+0000 7f7f437d5700 1 -- 172.21.15.113:0/1174462353 wait complete. 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.798+0000 7f7f437d5700 1 Processor -- start 2024-01-16T13:52:08.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.798+0000 7f7f437d5700 1 -- start start 2024-01-16T13:52:08.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28003a00 0x7f7f3c1519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 0x7f7f3c0ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3c0ab790 0x7f7f3c0ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0abd10 con 0x7f7f3c14ed80 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0abff0 con 0x7f7f3c0ab790 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0cafd0 con 0x7f7f28003a00 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 0x7f7f3c0ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 0x7f7f3c0ab250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40328/0 (socket says 172.21.15.113:40328) 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 -- 172.21.15.113:0/714775555 learned_addr learned my addr 172.21.15.113:0/714775555 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f41571700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28003a00 0x7f7f3c1519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f41d72700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3c0ab790 0x7f7f3c0ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 -- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28003a00 msgr2=0x7f7f3c1519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28003a00 0x7f7f3c1519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 -- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3c0ab790 msgr2=0x7f7f3c0ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3c0ab790 0x7f7f3c0ac500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 -- 172.21.15.113:0/714775555 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0cb2b0 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f40d70700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 0x7f7f3c0ab250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00e460 tx=0x7f7f2c00e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0050d0 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c005230 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c003680 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c0cb590 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.799+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c0aca40 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.800+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c0cb7d0 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.801+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7f2c0037e0 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.801+0000 7f7f31ffb700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.801+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f2c089000 con 0x7f7f3c14ed80 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.801+0000 7f7f41571700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:08.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.802+0000 7f7f41571700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f3801e040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:08.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.806+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f3c0cb7d0 con 0x7f7f3c14ed80 2024-01-16T13:52:08.847 INFO:tasks.ceph.mds.b.smithi124.stderr:2024-01-16T13:52:08.865+0000 7f0341dfa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T13:52:08.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:08.997+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f3c0d2c30 con 0x7f7f2006b350 2024-01-16T13:52:08.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.004+0000 7f7f31ffb700 1 -- 172.21.15.113:0/714775555 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264716 (secure 0 0 0) 0x7f7f38006c60 con 0x7f7f2006b350 2024-01-16T13:52:08.990 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 msgr2=0x7f7f2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7f38000c00 tx=0x7f7f3801e040 comp rx=0 tx=0).stop 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 msgr2=0x7f7f3c0ab250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 0x7f7f3c0ab250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00e460 tx=0x7f7f2c00e770 comp rx=0 tx=0).stop 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 shutdown_connections 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3c0ab790 0x7f7f3c0ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f28003a00 0x7f7f3c1519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 --2- 172.21.15.113:0/714775555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f3c14ed80 0x7f7f3c0ab250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:08.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 >> 172.21.15.113:0/714775555 conn(0x7f7f3c0b5b50 msgr2=0x7f7f3c15d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:08.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 shutdown_connections 2024-01-16T13:52:08.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.007+0000 7f7f437d5700 1 -- 172.21.15.113:0/714775555 wait complete. 2024-01-16T13:52:08.993 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:52:09.001 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-01-16T13:52:08.402241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.516646"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383364+0000","last_change":"2024-01-16T13:52:04.933545+0000","last_active":"2024-01-16T13:52:06.383364+0000","last_peered":"2024-01-16T13:52:06.383364+0000","last_clean":"2024-01-16T13:52:06.383364+0000","last_became_active":"2024-01-16T13:52:04.932314+0000","last_became_peered":"2024-01-16T13:52:04.932314+0000","last_unstale":"2024-01-16T13:52:06.383364+0000","last_undegraded":"2024-01-16T13:52:06.383364+0000","last_fullsized":"2024-01-16T13:52:06.383364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.917385+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.916958+0000","last_became_peered":"2024-01-16T13:52:04.916958+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382533+0000","last_change":"2024-01-16T13:52:04.917514+0000","last_active":"2024-01-16T13:52:06.382533+0000","last_peered":"2024-01-16T13:52:06.382533+0000","last_clean":"2024-01-16T13:52:06.382533+0000","last_became_active":"2024-01-16T13:52:04.916822+0000","last_became_peered":"2024-01-16T13:52:04.916822+0000","last_unstale":"2024-01-16T13:52:06.382533+0000","last_undegraded":"2024-01-16T13:52:06.382533+0000","last_fullsized":"2024-01-16T13:52:06.382533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:05.397603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904528+0000","last_change":"2024-01-16T13:52:04.926700+0000","last_active":"2024-01-16T13:52:05.904528+0000","last_peered":"2024-01-16T13:52:05.904528+0000","last_clean":"2024-01-16T13:52:05.904528+0000","last_became_active":"2024-01-16T13:52:04.923728+0000","last_became_peered":"2024-01-16T13:52:04.923728+0000","last_unstale":"2024-01-16T13:52:05.904528+0000","last_undegraded":"2024-01-16T13:52:05.904528+0000","last_fullsized":"2024-01-16T13:52:05.904528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382807+0000","last_change":"2024-01-16T13:52:04.931697+0000","last_active":"2024-01-16T13:52:06.382807+0000","last_peered":"2024-01-16T13:52:06.382807+0000","last_clean":"2024-01-16T13:52:06.382807+0000","last_became_active":"2024-01-16T13:52:04.931415+0000","last_became_peered":"2024-01-16T13:52:04.931415+0000","last_unstale":"2024-01-16T13:52:06.382807+0000","last_undegraded":"2024-01-16T13:52:06.382807+0000","last_fullsized":"2024-01-16T13:52:06.382807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904991+0000","last_change":"2024-01-16T13:52:04.925527+0000","last_active":"2024-01-16T13:52:05.904991+0000","last_peered":"2024-01-16T13:52:05.904991+0000","last_clean":"2024-01-16T13:52:05.904991+0000","last_became_active":"2024-01-16T13:52:04.923026+0000","last_became_peered":"2024-01-16T13:52:04.923026+0000","last_unstale":"2024-01-16T13:52:05.904991+0000","last_undegraded":"2024-01-16T13:52:05.904991+0000","last_fullsized":"2024-01-16T13:52:05.904991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382610+0000","last_change":"2024-01-16T13:52:04.930095+0000","last_active":"2024-01-16T13:52:06.382610+0000","last_peered":"2024-01-16T13:52:06.382610+0000","last_clean":"2024-01-16T13:52:06.382610+0000","last_became_active":"2024-01-16T13:52:04.926632+0000","last_became_peered":"2024-01-16T13:52:04.926632+0000","last_unstale":"2024-01-16T13:52:06.382610+0000","last_undegraded":"2024-01-16T13:52:06.382610+0000","last_fullsized":"2024-01-16T13:52:06.382610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382842+0000","last_change":"2024-01-16T13:52:04.927129+0000","last_active":"2024-01-16T13:52:06.382842+0000","last_peered":"2024-01-16T13:52:06.382842+0000","last_clean":"2024-01-16T13:52:06.382842+0000","last_became_active":"2024-01-16T13:52:04.926541+0000","last_became_peered":"2024-01-16T13:52:04.926541+0000","last_unstale":"2024-01-16T13:52:06.382842+0000","last_undegraded":"2024-01-16T13:52:06.382842+0000","last_fullsized":"2024-01-16T13:52:06.382842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383203+0000","last_change":"2024-01-16T13:52:04.935989+0000","last_active":"2024-01-16T13:52:06.383203+0000","last_peered":"2024-01-16T13:52:06.383203+0000","last_clean":"2024-01-16T13:52:06.383203+0000","last_became_active":"2024-01-16T13:52:04.935750+0000","last_became_peered":"2024-01-16T13:52:04.935750+0000","last_unstale":"2024-01-16T13:52:06.383203+0000","last_undegraded":"2024-01-16T13:52:06.383203+0000","last_fullsized":"2024-01-16T13:52:06.383203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904269+0000","last_change":"2024-01-16T13:52:04.941805+0000","last_active":"2024-01-16T13:52:05.904269+0000","last_peered":"2024-01-16T13:52:05.904269+0000","last_clean":"2024-01-16T13:52:05.904269+0000","last_became_active":"2024-01-16T13:52:04.941552+0000","last_became_peered":"2024-01-16T13:52:04.941552+0000","last_unstale":"2024-01-16T13:52:05.904269+0000","last_undegraded":"2024-01-16T13:52:05.904269+0000","last_fullsized":"2024-01-16T13:52:05.904269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904181+0000","last_change":"2024-01-16T13:52:04.941357+0000","last_active":"2024-01-16T13:52:05.904181+0000","last_peered":"2024-01-16T13:52:05.904181+0000","last_clean":"2024-01-16T13:52:05.904181+0000","last_became_active":"2024-01-16T13:52:04.940946+0000","last_became_peered":"2024-01-16T13:52:04.940946+0000","last_unstale":"2024-01-16T13:52:05.904181+0000","last_undegraded":"2024-01-16T13:52:05.904181+0000","last_fullsized":"2024-01-16T13:52:05.904181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383260+0000","last_change":"2024-01-16T13:52:04.923980+0000","last_active":"2024-01-16T13:52:06.383260+0000","last_peered":"2024-01-16T13:52:06.383260+0000","last_clean":"2024-01-16T13:52:06.383260+0000","last_became_active":"2024-01-16T13:52:04.923398+0000","last_became_peered":"2024-01-16T13:52:04.923398+0000","last_unstale":"2024-01-16T13:52:06.383260+0000","last_undegraded":"2024-01-16T13:52:06.383260+0000","last_fullsized":"2024-01-16T13:52:06.383260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383179+0000","last_change":"2024-01-16T13:52:04.922440+0000","last_active":"2024-01-16T13:52:06.383179+0000","last_peered":"2024-01-16T13:52:06.383179+0000","last_clean":"2024-01-16T13:52:06.383179+0000","last_became_active":"2024-01-16T13:52:04.922170+0000","last_became_peered":"2024-01-16T13:52:04.922170+0000","last_unstale":"2024-01-16T13:52:06.383179+0000","last_undegraded":"2024-01-16T13:52:06.383179+0000","last_fullsized":"2024-01-16T13:52:06.383179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382935+0000","last_change":"2024-01-16T13:52:04.918126+0000","last_active":"2024-01-16T13:52:06.382935+0000","last_peered":"2024-01-16T13:52:06.382935+0000","last_clean":"2024-01-16T13:52:06.382935+0000","last_became_active":"2024-01-16T13:52:04.917683+0000","last_became_peered":"2024-01-16T13:52:04.917683+0000","last_unstale":"2024-01-16T13:52:06.382935+0000","last_undegraded":"2024-01-16T13:52:06.382935+0000","last_fullsized":"2024-01-16T13:52:06.382935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904662+0000","last_change":"2024-01-16T13:52:04.919145+0000","last_active":"2024-01-16T13:52:05.904662+0000","last_peered":"2024-01-16T13:52:05.904662+0000","last_clean":"2024-01-16T13:52:05.904662+0000","last_became_active":"2024-01-16T13:52:04.918897+0000","last_became_peered":"2024-01-16T13:52:04.918897+0000","last_unstale":"2024-01-16T13:52:05.904662+0000","last_undegraded":"2024-01-16T13:52:05.904662+0000","last_fullsized":"2024-01-16T13:52:05.904662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.923697+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.923053+0000","last_became_peered":"2024-01-16T13:52:04.923053+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904261+0000","last_change":"2024-01-16T13:52:04.927671+0000","last_active":"2024-01-16T13:52:05.904261+0000","last_peered":"2024-01-16T13:52:05.904261+0000","last_clean":"2024-01-16T13:52:05.904261+0000","last_became_active":"2024-01-16T13:52:04.927402+0000","last_became_peered":"2024-01-16T13:52:04.927402+0000","last_unstale":"2024-01-16T13:52:05.904261+0000","last_undegraded":"2024-01-16T13:52:05.904261+0000","last_fullsized":"2024-01-16T13:52:05.904261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382408+0000","last_change":"2024-01-16T13:52:04.924972+0000","last_active":"2024-01-16T13:52:06.382408+0000","last_peered":"2024-01-16T13:52:06.382408+0000","last_clean":"2024-01-16T13:52:06.382408+0000","last_became_active":"2024-01-16T13:52:04.924756+0000","last_became_peered":"2024-01-16T13:52:04.924756+0000","last_unstale":"2024-01-16T13:52:06.382408+0000","last_undegraded":"2024-01-16T13:52:06.382408+0000","last_fullsized":"2024-01-16T13:52:06.382408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383026+0000","last_change":"2024-01-16T13:52:04.932352+0000","last_active":"2024-01-16T13:52:06.383026+0000","last_peered":"2024-01-16T13:52:06.383026+0000","last_clean":"2024-01-16T13:52:06.383026+0000","last_became_active":"2024-01-16T13:52:04.926615+0000","last_became_peered":"2024-01-16T13:52:04.926615+0000","last_unstale":"2024-01-16T13:52:06.383026+0000","last_undegraded":"2024-01-16T13:52:06.383026+0000","last_fullsized":"2024-01-16T13:52:06.383026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382783+0000","last_change":"2024-01-16T13:52:03.906795+0000","last_active":"2024-01-16T13:52:06.382783+0000","last_peered":"2024-01-16T13:52:06.382783+0000","last_clean":"2024-01-16T13:52:06.382783+0000","last_became_active":"2024-01-16T13:52:03.906413+0000","last_became_peered":"2024-01-16T13:52:03.906413+0000","last_unstale":"2024-01-16T13:52:06.382783+0000","last_undegraded":"2024-01-16T13:52:06.382783+0000","last_fullsized":"2024-01-16T13:52:06.382783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383205+0000","last_change":"2024-01-16T13:52:04.922947+0000","last_active":"2024-01-16T13:52:06.383205+0000","last_peered":"2024-01-16T13:52:06.383205+0000","last_clean":"2024-01-16T13:52:06.383205+0000","last_became_active":"2024-01-16T13:52:04.922641+0000","last_became_peered":"2024-01-16T13:52:04.922641+0000","last_unstale":"2024-01-16T13:52:06.383205+0000","last_undegraded":"2024-01-16T13:52:06.383205+0000","last_fullsized":"2024-01-16T13:52:06.383205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383275+0000","last_change":"2024-01-16T13:52:03.911364+0000","last_active":"2024-01-16T13:52:06.383275+0000","last_peered":"2024-01-16T13:52:06.383275+0000","last_clean":"2024-01-16T13:52:06.383275+0000","last_became_active":"2024-01-16T13:52:03.911096+0000","last_became_peered":"2024-01-16T13:52:03.911096+0000","last_unstale":"2024-01-16T13:52:06.383275+0000","last_undegraded":"2024-01-16T13:52:06.383275+0000","last_fullsized":"2024-01-16T13:52:06.383275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382855+0000","last_change":"2024-01-16T13:52:04.941583+0000","last_active":"2024-01-16T13:52:06.382855+0000","last_peered":"2024-01-16T13:52:06.382855+0000","last_clean":"2024-01-16T13:52:06.382855+0000","last_became_active":"2024-01-16T13:52:04.941270+0000","last_became_peered":"2024-01-16T13:52:04.941270+0000","last_unstale":"2024-01-16T13:52:06.382855+0000","last_undegraded":"2024-01-16T13:52:06.382855+0000","last_fullsized":"2024-01-16T13:52:06.382855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383462+0000","last_change":"2024-01-16T13:52:03.908013+0000","last_active":"2024-01-16T13:52:06.383462+0000","last_peered":"2024-01-16T13:52:06.383462+0000","last_clean":"2024-01-16T13:52:06.383462+0000","last_became_active":"2024-01-16T13:52:03.907716+0000","last_became_peered":"2024-01-16T13:52:03.907716+0000","last_unstale":"2024-01-16T13:52:06.383462+0000","last_undegraded":"2024-01-16T13:52:06.383462+0000","last_fullsized":"2024-01-16T13:52:06.383462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904017+0000","last_change":"2024-01-16T13:52:04.920391+0000","last_active":"2024-01-16T13:52:05.904017+0000","last_peered":"2024-01-16T13:52:05.904017+0000","last_clean":"2024-01-16T13:52:05.904017+0000","last_became_active":"2024-01-16T13:52:04.920118+0000","last_became_peered":"2024-01-16T13:52:04.920118+0000","last_unstale":"2024-01-16T13:52:05.904017+0000","last_undegraded":"2024-01-16T13:52:05.904017+0000","last_fullsized":"2024-01-16T13:52:05.904017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382906+0000","last_change":"2024-01-16T13:52:03.905310+0000","last_active":"2024-01-16T13:52:06.382906+0000","last_peered":"2024-01-16T13:52:06.382906+0000","last_clean":"2024-01-16T13:52:06.382906+0000","last_became_active":"2024-01-16T13:52:03.905031+0000","last_became_peered":"2024-01-16T13:52:03.905031+0000","last_unstale":"2024-01-16T13:52:06.382906+0000","last_undegraded":"2024-01-16T13:52:06.382906+0000","last_fullsized":"2024-01-16T13:52:06.382906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904705+0000","last_change":"2024-01-16T13:52:04.912084+0000","last_active":"2024-01-16T13:52:05.904705+0000","last_peered":"2024-01-16T13:52:05.904705+0000","last_clean":"2024-01-16T13:52:05.904705+0000","last_became_active":"2024-01-16T13:52:04.911792+0000","last_became_peered":"2024-01-16T13:52:04.911792+0000","last_unstale":"2024-01-16T13:52:05.904705+0000","last_undegraded":"2024-01-16T13:52:05.904705+0000","last_fullsized":"2024-01-16T13:52:05.904705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383224+0000","last_change":"2024-01-16T13:52:03.912985+0000","last_active":"2024-01-16T13:52:06.383224+0000","last_peered":"2024-01-16T13:52:06.383224+0000","last_clean":"2024-01-16T13:52:06.383224+0000","last_became_active":"2024-01-16T13:52:03.912758+0000","last_became_peered":"2024-01-16T13:52:03.912758+0000","last_unstale":"2024-01-16T13:52:06.383224+0000","last_undegraded":"2024-01-16T13:52:06.383224+0000","last_fullsized":"2024-01-16T13:52:06.383224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382525+0000","last_change":"2024-01-16T13:52:04.917347+0000","last_active":"2024-01-16T13:52:06.382525+0000","last_peered":"2024-01-16T13:52:06.382525+0000","last_clean":"2024-01-16T13:52:06.382525+0000","last_became_active":"2024-01-16T13:52:04.916918+0000","last_became_peered":"2024-01-16T13:52:04.916918+0000","last_unstale":"2024-01-16T13:52:06.382525+0000","last_undegraded":"2024-01-16T13:52:06.382525+0000","last_fullsized":"2024-01-16T13:52:06.382525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:51.832998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903890+0000","last_change":"2024-01-16T13:52:03.903840+0000","last_active":"2024-01-16T13:52:05.903890+0000","last_peered":"2024-01-16T13:52:05.903890+0000","last_clean":"2024-01-16T13:52:05.903890+0000","last_became_active":"2024-01-16T13:52:03.903524+0000","last_became_peered":"2024-01-16T13:52:03.903524+0000","last_unstale":"2024-01-16T13:52:05.903890+0000","last_undegraded":"2024-01-16T13:52:05.903890+0000","last_fullsized":"2024-01-16T13:52:05.903890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383068+0000","last_change":"2024-01-16T13:52:04.924992+0000","last_active":"2024-01-16T13:52:06.383068+0000","last_peered":"2024-01-16T13:52:06.383068+0000","last_clean":"2024-01-16T13:52:06.383068+0000","last_became_active":"2024-01-16T13:52:04.924689+0000","last_became_peered":"2024-01-16T13:52:04.924689+0000","last_unstale":"2024-01-16T13:52:06.383068+0000","last_undegraded":"2024-01-16T13:52:06.383068+0000","last_fullsized":"2024-01-16T13:52:06.383068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383042+0000","last_change":"2024-01-16T13:52:03.912658+0000","last_active":"2024-01-16T13:52:06.383042+0000","last_peered":"2024-01-16T13:52:06.383042+0000","last_clean":"2024-01-16T13:52:06.383042+0000","last_became_active":"2024-01-16T13:52:03.912382+0000","last_became_peered":"2024-01-16T13:52:03.912382+0000","last_unstale":"2024-01-16T13:52:06.383042+0000","last_undegraded":"2024-01-16T13:52:06.383042+0000","last_fullsized":"2024-01-16T13:52:06.383042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383509+0000","last_change":"2024-01-16T13:52:04.920332+0000","last_active":"2024-01-16T13:52:06.383509+0000","last_peered":"2024-01-16T13:52:06.383509+0000","last_clean":"2024-01-16T13:52:06.383509+0000","last_became_active":"2024-01-16T13:52:04.920061+0000","last_became_peered":"2024-01-16T13:52:04.920061+0000","last_unstale":"2024-01-16T13:52:06.383509+0000","last_undegraded":"2024-01-16T13:52:06.383509+0000","last_fullsized":"2024-01-16T13:52:06.383509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904985+0000","last_change":"2024-01-16T13:52:03.897543+0000","last_active":"2024-01-16T13:52:05.904985+0000","last_peered":"2024-01-16T13:52:05.904985+0000","last_clean":"2024-01-16T13:52:05.904985+0000","last_became_active":"2024-01-16T13:52:03.897059+0000","last_became_peered":"2024-01-16T13:52:03.897059+0000","last_unstale":"2024-01-16T13:52:05.904985+0000","last_undegraded":"2024-01-16T13:52:05.904985+0000","last_fullsized":"2024-01-16T13:52:05.904985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904732+0000","last_change":"2024-01-16T13:52:04.906828+0000","last_active":"2024-01-16T13:52:05.904732+0000","last_peered":"2024-01-16T13:52:05.904732+0000","last_clean":"2024-01-16T13:52:05.904732+0000","last_became_active":"2024-01-16T13:52:04.906549+0000","last_became_peered":"2024-01-16T13:52:04.906549+0000","last_unstale":"2024-01-16T13:52:05.904732+0000","last_undegraded":"2024-01-16T13:52:05.904732+0000","last_fullsized":"2024-01-16T13:52:05.904732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904051+0000","last_change":"2024-01-16T13:52:03.901711+0000","last_active":"2024-01-16T13:52:05.904051+0000","last_peered":"2024-01-16T13:52:05.904051+0000","last_clean":"2024-01-16T13:52:05.904051+0000","last_became_active":"2024-01-16T13:52:03.901420+0000","last_became_peered":"2024-01-16T13:52:03.901420+0000","last_unstale":"2024-01-16T13:52:05.904051+0000","last_undegraded":"2024-01-16T13:52:05.904051+0000","last_fullsized":"2024-01-16T13:52:05.904051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904698+0000","last_change":"2024-01-16T13:52:04.935326+0000","last_active":"2024-01-16T13:52:05.904698+0000","last_peered":"2024-01-16T13:52:05.904698+0000","last_clean":"2024-01-16T13:52:05.904698+0000","last_became_active":"2024-01-16T13:52:04.934628+0000","last_became_peered":"2024-01-16T13:52:04.934628+0000","last_unstale":"2024-01-16T13:52:05.904698+0000","last_undegraded":"2024-01-16T13:52:05.904698+0000","last_fullsized":"2024-01-16T13:52:05.904698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904576+0000","last_change":"2024-01-16T13:52:03.902616+0000","last_active":"2024-01-16T13:52:05.904576+0000","last_peered":"2024-01-16T13:52:05.904576+0000","last_clean":"2024-01-16T13:52:05.904576+0000","last_became_active":"2024-01-16T13:52:03.902333+0000","last_became_peered":"2024-01-16T13:52:03.902333+0000","last_unstale":"2024-01-16T13:52:05.904576+0000","last_undegraded":"2024-01-16T13:52:05.904576+0000","last_fullsized":"2024-01-16T13:52:05.904576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383250+0000","last_change":"2024-01-16T13:52:04.936172+0000","last_active":"2024-01-16T13:52:06.383250+0000","last_peered":"2024-01-16T13:52:06.383250+0000","last_clean":"2024-01-16T13:52:06.383250+0000","last_became_active":"2024-01-16T13:52:04.935881+0000","last_became_peered":"2024-01-16T13:52:04.935881+0000","last_unstale":"2024-01-16T13:52:06.383250+0000","last_undegraded":"2024-01-16T13:52:06.383250+0000","last_fullsized":"2024-01-16T13:52:06.383250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904808+0000","last_change":"2024-01-16T13:52:03.896280+0000","last_active":"2024-01-16T13:52:05.904808+0000","last_peered":"2024-01-16T13:52:05.904808+0000","last_clean":"2024-01-16T13:52:05.904808+0000","last_became_active":"2024-01-16T13:52:03.896001+0000","last_became_peered":"2024-01-16T13:52:03.896001+0000","last_unstale":"2024-01-16T13:52:05.904808+0000","last_undegraded":"2024-01-16T13:52:05.904808+0000","last_fullsized":"2024-01-16T13:52:05.904808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383116+0000","last_change":"2024-01-16T13:52:04.927368+0000","last_active":"2024-01-16T13:52:06.383116+0000","last_peered":"2024-01-16T13:52:06.383116+0000","last_clean":"2024-01-16T13:52:06.383116+0000","last_became_active":"2024-01-16T13:52:04.927100+0000","last_became_peered":"2024-01-16T13:52:04.927100+0000","last_unstale":"2024-01-16T13:52:06.383116+0000","last_undegraded":"2024-01-16T13:52:06.383116+0000","last_fullsized":"2024-01-16T13:52:06.383116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904369+0000","last_change":"2024-01-16T13:52:03.910360+0000","last_active":"2024-01-16T13:52:05.904369+0000","last_peered":"2024-01-16T13:52:05.904369+0000","last_clean":"2024-01-16T13:52:05.904369+0000","last_became_active":"2024-01-16T13:52:03.910098+0000","last_became_peered":"2024-01-16T13:52:03.910098+0000","last_unstale":"2024-01-16T13:52:05.904369+0000","last_undegraded":"2024-01-16T13:52:05.904369+0000","last_fullsized":"2024-01-16T13:52:05.904369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383195+0000","last_change":"2024-01-16T13:52:04.932597+0000","last_active":"2024-01-16T13:52:06.383195+0000","last_peered":"2024-01-16T13:52:06.383195+0000","last_clean":"2024-01-16T13:52:06.383195+0000","last_became_active":"2024-01-16T13:52:04.930511+0000","last_became_peered":"2024-01-16T13:52:04.930511+0000","last_unstale":"2024-01-16T13:52:06.383195+0000","last_undegraded":"2024-01-16T13:52:06.383195+0000","last_fullsized":"2024-01-16T13:52:06.383195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382997+0000","last_change":"2024-01-16T13:52:03.906336+0000","last_active":"2024-01-16T13:52:06.382997+0000","last_peered":"2024-01-16T13:52:06.382997+0000","last_clean":"2024-01-16T13:52:06.382997+0000","last_became_active":"2024-01-16T13:52:03.905656+0000","last_became_peered":"2024-01-16T13:52:03.905656+0000","last_unstale":"2024-01-16T13:52:06.382997+0000","last_undegraded":"2024-01-16T13:52:06.382997+0000","last_fullsized":"2024-01-16T13:52:06.382997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382700+0000","last_change":"2024-01-16T13:52:04.927208+0000","last_active":"2024-01-16T13:52:06.382700+0000","last_peered":"2024-01-16T13:52:06.382700+0000","last_clean":"2024-01-16T13:52:06.382700+0000","last_became_active":"2024-01-16T13:52:04.926931+0000","last_became_peered":"2024-01-16T13:52:04.926931+0000","last_unstale":"2024-01-16T13:52:06.382700+0000","last_undegraded":"2024-01-16T13:52:06.382700+0000","last_fullsized":"2024-01-16T13:52:06.382700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905228+0000","last_change":"2024-01-16T13:52:03.911290+0000","last_active":"2024-01-16T13:52:05.905228+0000","last_peered":"2024-01-16T13:52:05.905228+0000","last_clean":"2024-01-16T13:52:05.905228+0000","last_became_active":"2024-01-16T13:52:03.911029+0000","last_became_peered":"2024-01-16T13:52:03.911029+0000","last_unstale":"2024-01-16T13:52:05.905228+0000","last_undegraded":"2024-01-16T13:52:05.905228+0000","last_fullsized":"2024-01-16T13:52:05.905228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:40.789717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383244+0000","last_change":"2024-01-16T13:52:04.932349+0000","last_active":"2024-01-16T13:52:06.383244+0000","last_peered":"2024-01-16T13:52:06.383244+0000","last_clean":"2024-01-16T13:52:06.383244+0000","last_became_active":"2024-01-16T13:52:04.931854+0000","last_became_peered":"2024-01-16T13:52:04.931854+0000","last_unstale":"2024-01-16T13:52:06.383244+0000","last_undegraded":"2024-01-16T13:52:06.383244+0000","last_fullsized":"2024-01-16T13:52:06.383244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903653+0000","last_change":"2024-01-16T13:52:03.903815+0000","last_active":"2024-01-16T13:52:05.903653+0000","last_peered":"2024-01-16T13:52:05.903653+0000","last_clean":"2024-01-16T13:52:05.903653+0000","last_became_active":"2024-01-16T13:52:03.903487+0000","last_became_peered":"2024-01-16T13:52:03.903487+0000","last_unstale":"2024-01-16T13:52:05.903653+0000","last_undegraded":"2024-01-16T13:52:05.903653+0000","last_fullsized":"2024-01-16T13:52:05.903653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:49.732984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903921+0000","last_change":"2024-01-16T13:52:04.922443+0000","last_active":"2024-01-16T13:52:05.903921+0000","last_peered":"2024-01-16T13:52:05.903921+0000","last_clean":"2024-01-16T13:52:05.903921+0000","last_became_active":"2024-01-16T13:52:04.922131+0000","last_became_peered":"2024-01-16T13:52:04.922131+0000","last_unstale":"2024-01-16T13:52:05.903921+0000","last_undegraded":"2024-01-16T13:52:05.903921+0000","last_fullsized":"2024-01-16T13:52:05.903921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382223+0000","last_change":"2024-01-16T13:52:03.910304+0000","last_active":"2024-01-16T13:52:06.382223+0000","last_peered":"2024-01-16T13:52:06.382223+0000","last_clean":"2024-01-16T13:52:06.382223+0000","last_became_active":"2024-01-16T13:52:03.910009+0000","last_became_peered":"2024-01-16T13:52:03.910009+0000","last_unstale":"2024-01-16T13:52:06.382223+0000","last_undegraded":"2024-01-16T13:52:06.382223+0000","last_fullsized":"2024-01-16T13:52:06.382223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382979+0000","last_change":"2024-01-16T13:52:04.930880+0000","last_active":"2024-01-16T13:52:06.382979+0000","last_peered":"2024-01-16T13:52:06.382979+0000","last_clean":"2024-01-16T13:52:06.382979+0000","last_became_active":"2024-01-16T13:52:04.930553+0000","last_became_peered":"2024-01-16T13:52:04.930553+0000","last_unstale":"2024-01-16T13:52:06.382979+0000","last_undegraded":"2024-01-16T13:52:06.382979+0000","last_fullsized":"2024-01-16T13:52:06.382979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382946+0000","last_change":"2024-01-16T13:52:03.906573+0000","last_active":"2024-01-16T13:52:06.382946+0000","last_peered":"2024-01-16T13:52:06.382946+0000","last_clean":"2024-01-16T13:52:06.382946+0000","last_became_active":"2024-01-16T13:52:03.906257+0000","last_became_peered":"2024-01-16T13:52:03.906257+0000","last_unstale":"2024-01-16T13:52:06.382946+0000","last_undegraded":"2024-01-16T13:52:06.382946+0000","last_fullsized":"2024-01-16T13:52:06.382946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:14:03.506561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382905+0000","last_change":"2024-01-16T13:52:04.931581+0000","last_active":"2024-01-16T13:52:06.382905+0000","last_peered":"2024-01-16T13:52:06.382905+0000","last_clean":"2024-01-16T13:52:06.382905+0000","last_became_active":"2024-01-16T13:52:04.931306+0000","last_became_peered":"2024-01-16T13:52:04.931306+0000","last_unstale":"2024-01-16T13:52:06.382905+0000","last_undegraded":"2024-01-16T13:52:06.382905+0000","last_fullsized":"2024-01-16T13:52:06.382905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904968+0000","last_change":"2024-01-16T13:52:03.903873+0000","last_active":"2024-01-16T13:52:05.904968+0000","last_peered":"2024-01-16T13:52:05.904968+0000","last_clean":"2024-01-16T13:52:05.904968+0000","last_became_active":"2024-01-16T13:52:03.903582+0000","last_became_peered":"2024-01-16T13:52:03.903582+0000","last_unstale":"2024-01-16T13:52:05.904968+0000","last_undegraded":"2024-01-16T13:52:05.904968+0000","last_fullsized":"2024-01-16T13:52:05.904968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904494+0000","last_change":"2024-01-16T13:52:04.926087+0000","last_active":"2024-01-16T13:52:05.904494+0000","last_peered":"2024-01-16T13:52:05.904494+0000","last_clean":"2024-01-16T13:52:05.904494+0000","last_became_active":"2024-01-16T13:52:04.925736+0000","last_became_peered":"2024-01-16T13:52:04.925736+0000","last_unstale":"2024-01-16T13:52:05.904494+0000","last_undegraded":"2024-01-16T13:52:05.904494+0000","last_fullsized":"2024-01-16T13:52:05.904494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905152+0000","last_change":"2024-01-16T13:52:03.901400+0000","last_active":"2024-01-16T13:52:05.905152+0000","last_peered":"2024-01-16T13:52:05.905152+0000","last_clean":"2024-01-16T13:52:05.905152+0000","last_became_active":"2024-01-16T13:52:03.901087+0000","last_became_peered":"2024-01-16T13:52:03.901087+0000","last_unstale":"2024-01-16T13:52:05.905152+0000","last_undegraded":"2024-01-16T13:52:05.905152+0000","last_fullsized":"2024-01-16T13:52:05.905152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:43.211649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382982+0000","last_change":"2024-01-16T13:52:04.917916+0000","last_active":"2024-01-16T13:52:06.382982+0000","last_peered":"2024-01-16T13:52:06.382982+0000","last_clean":"2024-01-16T13:52:06.382982+0000","last_became_active":"2024-01-16T13:52:04.917649+0000","last_became_peered":"2024-01-16T13:52:04.917649+0000","last_unstale":"2024-01-16T13:52:06.382982+0000","last_undegraded":"2024-01-16T13:52:06.382982+0000","last_fullsized":"2024-01-16T13:52:06.382982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:37.904615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904621+0000","last_change":"2024-01-16T13:52:04.916017+0000","last_active":"2024-01-16T13:52:05.904621+0000","last_peered":"2024-01-16T13:52:05.904621+0000","last_clean":"2024-01-16T13:52:05.904621+0000","last_became_active":"2024-01-16T13:52:04.915765+0000","last_became_peered":"2024-01-16T13:52:04.915765+0000","last_unstale":"2024-01-16T13:52:05.904621+0000","last_undegraded":"2024-01-16T13:52:05.904621+0000","last_fullsized":"2024-01-16T13:52:05.904621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382686+0000","last_change":"2024-01-16T13:52:03.910443+0000","last_active":"2024-01-16T13:52:06.382686+0000","last_peered":"2024-01-16T13:52:06.382686+0000","last_clean":"2024-01-16T13:52:06.382686+0000","last_became_active":"2024-01-16T13:52:03.910025+0000","last_became_peered":"2024-01-16T13:52:03.910025+0000","last_unstale":"2024-01-16T13:52:06.382686+0000","last_undegraded":"2024-01-16T13:52:06.382686+0000","last_fullsized":"2024-01-16T13:52:06.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:22.047669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383379+0000","last_change":"2024-01-16T13:52:04.927591+0000","last_active":"2024-01-16T13:52:06.383379+0000","last_peered":"2024-01-16T13:52:06.383379+0000","last_clean":"2024-01-16T13:52:06.383379+0000","last_became_active":"2024-01-16T13:52:04.927020+0000","last_became_peered":"2024-01-16T13:52:04.927020+0000","last_unstale":"2024-01-16T13:52:06.383379+0000","last_undegraded":"2024-01-16T13:52:06.383379+0000","last_fullsized":"2024-01-16T13:52:06.383379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903799+0000","last_change":"2024-01-16T13:52:03.906447+0000","last_active":"2024-01-16T13:52:05.903799+0000","last_peered":"2024-01-16T13:52:05.903799+0000","last_clean":"2024-01-16T13:52:05.903799+0000","last_became_active":"2024-01-16T13:52:03.906082+0000","last_became_peered":"2024-01-16T13:52:03.906082+0000","last_unstale":"2024-01-16T13:52:05.903799+0000","last_undegraded":"2024-01-16T13:52:05.903799+0000","last_fullsized":"2024-01-16T13:52:05.903799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T13:52:04.913338+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:04.913338+0000","last_peered":"2024-01-16T13:52:04.913338+0000","last_clean":"2024-01-16T13:52:04.913338+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:04.913338+0000","last_undegraded":"2024-01-16T13:52:04.913338+0000","last_fullsized":"2024-01-16T13:52:04.913338+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382590+0000","last_change":"2024-01-16T13:52:04.923544+0000","last_active":"2024-01-16T13:52:06.382590+0000","last_peered":"2024-01-16T13:52:06.382590+0000","last_clean":"2024-01-16T13:52:06.382590+0000","last_became_active":"2024-01-16T13:52:04.923283+0000","last_became_peered":"2024-01-16T13:52:04.923283+0000","last_unstale":"2024-01-16T13:52:06.382590+0000","last_undegraded":"2024-01-16T13:52:06.382590+0000","last_fullsized":"2024-01-16T13:52:06.382590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382524+0000","last_change":"2024-01-16T13:52:03.910495+0000","last_active":"2024-01-16T13:52:06.382524+0000","last_peered":"2024-01-16T13:52:06.382524+0000","last_clean":"2024-01-16T13:52:06.382524+0000","last_became_active":"2024-01-16T13:52:03.910162+0000","last_became_peered":"2024-01-16T13:52:03.910162+0000","last_unstale":"2024-01-16T13:52:06.382524+0000","last_undegraded":"2024-01-16T13:52:06.382524+0000","last_fullsized":"2024-01-16T13:52:06.382524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:44.922338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:03.910098+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:03.909843+0000","last_became_peered":"2024-01-16T13:52:03.909843+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904899+0000","last_change":"2024-01-16T13:52:04.915509+0000","last_active":"2024-01-16T13:52:05.904899+0000","last_peered":"2024-01-16T13:52:05.904899+0000","last_clean":"2024-01-16T13:52:05.904899+0000","last_became_active":"2024-01-16T13:52:04.915240+0000","last_became_peered":"2024-01-16T13:52:04.915240+0000","last_unstale":"2024-01-16T13:52:05.904899+0000","last_undegraded":"2024-01-16T13:52:05.904899+0000","last_fullsized":"2024-01-16T13:52:05.904899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904829+0000","last_change":"2024-01-16T13:52:03.903483+0000","last_active":"2024-01-16T13:52:05.904829+0000","last_peered":"2024-01-16T13:52:05.904829+0000","last_clean":"2024-01-16T13:52:05.904829+0000","last_became_active":"2024-01-16T13:52:03.903147+0000","last_became_peered":"2024-01-16T13:52:03.903147+0000","last_unstale":"2024-01-16T13:52:05.904829+0000","last_undegraded":"2024-01-16T13:52:05.904829+0000","last_fullsized":"2024-01-16T13:52:05.904829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383351+0000","last_change":"2024-01-16T13:52:04.927887+0000","last_active":"2024-01-16T13:52:06.383351+0000","last_peered":"2024-01-16T13:52:06.383351+0000","last_clean":"2024-01-16T13:52:06.383351+0000","last_became_active":"2024-01-16T13:52:04.927626+0000","last_became_peered":"2024-01-16T13:52:04.927626+0000","last_unstale":"2024-01-16T13:52:06.383351+0000","last_undegraded":"2024-01-16T13:52:06.383351+0000","last_fullsized":"2024-01-16T13:52:06.383351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904612+0000","last_change":"2024-01-16T13:52:03.903609+0000","last_active":"2024-01-16T13:52:05.904612+0000","last_peered":"2024-01-16T13:52:05.904612+0000","last_clean":"2024-01-16T13:52:05.904612+0000","last_became_active":"2024-01-16T13:52:03.903258+0000","last_became_peered":"2024-01-16T13:52:03.903258+0000","last_unstale":"2024-01-16T13:52:05.904612+0000","last_undegraded":"2024-01-16T13:52:05.904612+0000","last_fullsized":"2024-01-16T13:52:05.904612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383138+0000","last_change":"2024-01-16T13:52:04.918025+0000","last_active":"2024-01-16T13:52:06.383138+0000","last_peered":"2024-01-16T13:52:06.383138+0000","last_clean":"2024-01-16T13:52:06.383138+0000","last_became_active":"2024-01-16T13:52:04.917716+0000","last_became_peered":"2024-01-16T13:52:04.917716+0000","last_unstale":"2024-01-16T13:52:06.383138+0000","last_undegraded":"2024-01-16T13:52:06.383138+0000","last_fullsized":"2024-01-16T13:52:06.383138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382341+0000","last_change":"2024-01-16T13:52:03.909845+0000","last_active":"2024-01-16T13:52:06.382341+0000","last_peered":"2024-01-16T13:52:06.382341+0000","last_clean":"2024-01-16T13:52:06.382341+0000","last_became_active":"2024-01-16T13:52:03.909573+0000","last_became_peered":"2024-01-16T13:52:03.909573+0000","last_unstale":"2024-01-16T13:52:06.382341+0000","last_undegraded":"2024-01-16T13:52:06.382341+0000","last_fullsized":"2024-01-16T13:52:06.382341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:04.918497+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:04.918198+0000","last_became_peered":"2024-01-16T13:52:04.918198+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:01.775696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904593+0000","last_change":"2024-01-16T13:52:03.904013+0000","last_active":"2024-01-16T13:52:05.904593+0000","last_peered":"2024-01-16T13:52:05.904593+0000","last_clean":"2024-01-16T13:52:05.904593+0000","last_became_active":"2024-01-16T13:52:03.903827+0000","last_became_peered":"2024-01-16T13:52:03.903827+0000","last_unstale":"2024-01-16T13:52:05.904593+0000","last_undegraded":"2024-01-16T13:52:05.904593+0000","last_fullsized":"2024-01-16T13:52:05.904593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383603+0000","last_change":"2024-01-16T13:52:04.912859+0000","last_active":"2024-01-16T13:52:06.383603+0000","last_peered":"2024-01-16T13:52:06.383603+0000","last_clean":"2024-01-16T13:52:06.383603+0000","last_became_active":"2024-01-16T13:52:04.912597+0000","last_became_peered":"2024-01-16T13:52:04.912597+0000","last_unstale":"2024-01-16T13:52:06.383603+0000","last_undegraded":"2024-01-16T13:52:06.383603+0000","last_fullsized":"2024-01-16T13:52:06.383603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:03.908118+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:03.907800+0000","last_became_peered":"2024-01-16T13:52:03.907800+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383308+0000","last_change":"2024-01-16T13:52:04.918302+0000","last_active":"2024-01-16T13:52:06.383308+0000","last_peered":"2024-01-16T13:52:06.383308+0000","last_clean":"2024-01-16T13:52:06.383308+0000","last_became_active":"2024-01-16T13:52:04.917994+0000","last_became_peered":"2024-01-16T13:52:04.917994+0000","last_unstale":"2024-01-16T13:52:06.383308+0000","last_undegraded":"2024-01-16T13:52:06.383308+0000","last_fullsized":"2024-01-16T13:52:06.383308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383067+0000","last_change":"2024-01-16T13:52:03.903338+0000","last_active":"2024-01-16T13:52:06.383067+0000","last_peered":"2024-01-16T13:52:06.383067+0000","last_clean":"2024-01-16T13:52:06.383067+0000","last_became_active":"2024-01-16T13:52:03.903053+0000","last_became_peered":"2024-01-16T13:52:03.903053+0000","last_unstale":"2024-01-16T13:52:06.383067+0000","last_undegraded":"2024-01-16T13:52:06.383067+0000","last_fullsized":"2024-01-16T13:52:06.383067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:27.027544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903790+0000","last_change":"2024-01-16T13:52:04.924256+0000","last_active":"2024-01-16T13:52:05.903790+0000","last_peered":"2024-01-16T13:52:05.903790+0000","last_clean":"2024-01-16T13:52:05.903790+0000","last_became_active":"2024-01-16T13:52:04.923945+0000","last_became_peered":"2024-01-16T13:52:04.923945+0000","last_unstale":"2024-01-16T13:52:05.903790+0000","last_undegraded":"2024-01-16T13:52:05.903790+0000","last_fullsized":"2024-01-16T13:52:05.903790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382587+0000","last_change":"2024-01-16T13:52:03.906252+0000","last_active":"2024-01-16T13:52:06.382587+0000","last_peered":"2024-01-16T13:52:06.382587+0000","last_clean":"2024-01-16T13:52:06.382587+0000","last_became_active":"2024-01-16T13:52:03.905949+0000","last_became_peered":"2024-01-16T13:52:03.905949+0000","last_unstale":"2024-01-16T13:52:06.382587+0000","last_undegraded":"2024-01-16T13:52:06.382587+0000","last_fullsized":"2024-01-16T13:52:06.382587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:00.771273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:04.933410+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:04.932914+0000","last_became_peered":"2024-01-16T13:52:04.932914+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904753+0000","last_change":"2024-01-16T13:52:03.897651+0000","last_active":"2024-01-16T13:52:05.904753+0000","last_peered":"2024-01-16T13:52:05.904753+0000","last_clean":"2024-01-16T13:52:05.904753+0000","last_became_active":"2024-01-16T13:52:03.897380+0000","last_became_peered":"2024-01-16T13:52:03.897380+0000","last_unstale":"2024-01-16T13:52:05.904753+0000","last_undegraded":"2024-01-16T13:52:05.904753+0000","last_fullsized":"2024-01-16T13:52:05.904753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904179+0000","last_change":"2024-01-16T13:52:04.934873+0000","last_active":"2024-01-16T13:52:05.904179+0000","last_peered":"2024-01-16T13:52:05.904179+0000","last_clean":"2024-01-16T13:52:05.904179+0000","last_became_active":"2024-01-16T13:52:04.934573+0000","last_became_peered":"2024-01-16T13:52:04.934573+0000","last_unstale":"2024-01-16T13:52:05.904179+0000","last_undegraded":"2024-01-16T13:52:05.904179+0000","last_fullsized":"2024-01-16T13:52:05.904179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:03.911272+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:03.910949+0000","last_became_peered":"2024-01-16T13:52:03.910949+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383514+0000","last_change":"2024-01-16T13:52:04.933704+0000","last_active":"2024-01-16T13:52:06.383514+0000","last_peered":"2024-01-16T13:52:06.383514+0000","last_clean":"2024-01-16T13:52:06.383514+0000","last_became_active":"2024-01-16T13:52:04.933390+0000","last_became_peered":"2024-01-16T13:52:04.933390+0000","last_unstale":"2024-01-16T13:52:06.383514+0000","last_undegraded":"2024-01-16T13:52:06.383514+0000","last_fullsized":"2024-01-16T13:52:06.383514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903638+0000","last_change":"2024-01-16T13:52:04.922828+0000","last_active":"2024-01-16T13:52:05.903638+0000","last_peered":"2024-01-16T13:52:05.903638+0000","last_clean":"2024-01-16T13:52:05.903638+0000","last_became_active":"2024-01-16T13:52:04.922428+0000","last_became_peered":"2024-01-16T13:52:04.922428+0000","last_unstale":"2024-01-16T13:52:05.903638+0000","last_undegraded":"2024-01-16T13:52:05.903638+0000","last_fullsized":"2024-01-16T13:52:05.903638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382275+0000","last_change":"2024-01-16T13:52:04.915322+0000","last_active":"2024-01-16T13:52:06.382275+0000","last_peered":"2024-01-16T13:52:06.382275+0000","last_clean":"2024-01-16T13:52:06.382275+0000","last_became_active":"2024-01-16T13:52:04.915031+0000","last_became_peered":"2024-01-16T13:52:04.915031+0000","last_unstale":"2024-01-16T13:52:06.382275+0000","last_undegraded":"2024-01-16T13:52:06.382275+0000","last_fullsized":"2024-01-16T13:52:06.382275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905142+0000","last_change":"2024-01-16T13:52:04.925541+0000","last_active":"2024-01-16T13:52:05.905142+0000","last_peered":"2024-01-16T13:52:05.905142+0000","last_clean":"2024-01-16T13:52:05.905142+0000","last_became_active":"2024-01-16T13:52:04.923411+0000","last_became_peered":"2024-01-16T13:52:04.923411+0000","last_unstale":"2024-01-16T13:52:05.905142+0000","last_undegraded":"2024-01-16T13:52:05.905142+0000","last_fullsized":"2024-01-16T13:52:05.905142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382903+0000","last_change":"2024-01-16T13:52:04.919996+0000","last_active":"2024-01-16T13:52:06.382903+0000","last_peered":"2024-01-16T13:52:06.382903+0000","last_clean":"2024-01-16T13:52:06.382903+0000","last_became_active":"2024-01-16T13:52:04.919257+0000","last_became_peered":"2024-01-16T13:52:04.919257+0000","last_unstale":"2024-01-16T13:52:06.382903+0000","last_undegraded":"2024-01-16T13:52:06.382903+0000","last_fullsized":"2024-01-16T13:52:06.382903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382991+0000","last_change":"2024-01-16T13:52:04.921215+0000","last_active":"2024-01-16T13:52:06.382991+0000","last_peered":"2024-01-16T13:52:06.382991+0000","last_clean":"2024-01-16T13:52:06.382991+0000","last_became_active":"2024-01-16T13:52:04.920992+0000","last_became_peered":"2024-01-16T13:52:04.920992+0000","last_unstale":"2024-01-16T13:52:06.382991+0000","last_undegraded":"2024-01-16T13:52:06.382991+0000","last_fullsized":"2024-01-16T13:52:06.382991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:04.911541+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:04.911263+0000","last_became_peered":"2024-01-16T13:52:04.911263+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904058+0000","last_change":"2024-01-16T13:52:04.932056+0000","last_active":"2024-01-16T13:52:05.904058+0000","last_peered":"2024-01-16T13:52:05.904058+0000","last_clean":"2024-01-16T13:52:05.904058+0000","last_became_active":"2024-01-16T13:52:04.931794+0000","last_became_peered":"2024-01-16T13:52:04.931794+0000","last_unstale":"2024-01-16T13:52:05.904058+0000","last_undegraded":"2024-01-16T13:52:05.904058+0000","last_fullsized":"2024-01-16T13:52:05.904058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383217+0000","last_change":"2024-01-16T13:52:04.918735+0000","last_active":"2024-01-16T13:52:06.383217+0000","last_peered":"2024-01-16T13:52:06.383217+0000","last_clean":"2024-01-16T13:52:06.383217+0000","last_became_active":"2024-01-16T13:52:04.918440+0000","last_became_peered":"2024-01-16T13:52:04.918440+0000","last_unstale":"2024-01-16T13:52:06.383217+0000","last_undegraded":"2024-01-16T13:52:06.383217+0000","last_fullsized":"2024-01-16T13:52:06.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905101+0000","last_change":"2024-01-16T13:52:04.923904+0000","last_active":"2024-01-16T13:52:05.905101+0000","last_peered":"2024-01-16T13:52:05.905101+0000","last_clean":"2024-01-16T13:52:05.905101+0000","last_became_active":"2024-01-16T13:52:04.923426+0000","last_became_peered":"2024-01-16T13:52:04.923426+0000","last_unstale":"2024-01-16T13:52:05.905101+0000","last_undegraded":"2024-01-16T13:52:05.905101+0000","last_fullsized":"2024-01-16T13:52:05.905101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383063+0000","last_change":"2024-01-16T13:52:04.919553+0000","last_active":"2024-01-16T13:52:06.383063+0000","last_peered":"2024-01-16T13:52:06.383063+0000","last_clean":"2024-01-16T13:52:06.383063+0000","last_became_active":"2024-01-16T13:52:04.919053+0000","last_became_peered":"2024-01-16T13:52:04.919053+0000","last_unstale":"2024-01-16T13:52:06.383063+0000","last_undegraded":"2024-01-16T13:52:06.383063+0000","last_fullsized":"2024-01-16T13:52:06.383063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905086+0000","last_change":"2024-01-16T13:52:04.926131+0000","last_active":"2024-01-16T13:52:05.905086+0000","last_peered":"2024-01-16T13:52:05.905086+0000","last_clean":"2024-01-16T13:52:05.905086+0000","last_became_active":"2024-01-16T13:52:04.925848+0000","last_became_peered":"2024-01-16T13:52:04.925848+0000","last_unstale":"2024-01-16T13:52:05.905086+0000","last_undegraded":"2024-01-16T13:52:05.905086+0000","last_fullsized":"2024-01-16T13:52:05.905086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382745+0000","last_change":"2024-01-16T13:52:04.931968+0000","last_active":"2024-01-16T13:52:06.382745+0000","last_peered":"2024-01-16T13:52:06.382745+0000","last_clean":"2024-01-16T13:52:06.382745+0000","last_became_active":"2024-01-16T13:52:04.931678+0000","last_became_peered":"2024-01-16T13:52:04.931678+0000","last_unstale":"2024-01-16T13:52:06.382745+0000","last_undegraded":"2024-01-16T13:52:06.382745+0000","last_fullsized":"2024-01-16T13:52:06.382745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904558+0000","last_change":"2024-01-16T13:52:04.929411+0000","last_active":"2024-01-16T13:52:05.904558+0000","last_peered":"2024-01-16T13:52:05.904558+0000","last_clean":"2024-01-16T13:52:05.904558+0000","last_became_active":"2024-01-16T13:52:04.929143+0000","last_became_peered":"2024-01-16T13:52:04.929143+0000","last_unstale":"2024-01-16T13:52:05.904558+0000","last_undegraded":"2024-01-16T13:52:05.904558+0000","last_fullsized":"2024-01-16T13:52:05.904558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:52:09.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:52:09.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.181+0000 7fe3b0798700 1 Processor -- start 2024-01-16T13:52:09.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.181+0000 7fe3b0798700 1 -- start start 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3b0798700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe3a815e2a0 0x7fe3a806ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3b0798700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a806e7f0 con 0x7fe3a8073f90 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3b0798700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a806dec0 con 0x7fe3a80ce090 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3b0798700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a806e1a0 con 0x7fe3a815e2a0 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3ae534700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe3a815e2a0 0x7fe3a806ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3aed35700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe3a8073f90 0x7fe3a80cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51978/0 (socket says 172.21.15.113:51978) 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3aed35700 1 -- 172.21.15.113:0/2948997172 learned_addr learned my addr 172.21.15.113:0/2948997172 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3ae534700 1 -- 172.21.15.113:0/2948997172 >> v1:172.21.15.124:6789/0 conn(0x7fe3a80ce090 legacy=0x7fe3a806ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:09.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3ae534700 1 -- 172.21.15.113:0/2948997172 >> v1:172.21.15.113:6789/0 conn(0x7fe3a8073f90 legacy=0x7fe3a80cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3ae534700 1 -- 172.21.15.113:0/2948997172 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a806e480 con 0x7fe3a815e2a0 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.182+0000 7fe3ad532700 1 -- 172.21.15.113:0/2948997172 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229054653 0 0) 0x7fe3a806e7f0 con 0x7fe3a8073f90 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3ae534700 1 --2- 172.21.15.113:0/2948997172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a815e2a0 0x7fe3a806ba70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe398009840 tx=0x7fe398009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1428c4e7308bb7c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3ad532700 1 -- 172.21.15.113:0/2948997172 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398004230 con 0x7fe3a815e2a0 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3ad532700 1 -- 172.21.15.113:0/2948997172 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398004390 con 0x7fe3a815e2a0 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3ad532700 1 -- 172.21.15.113:0/2948997172 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980036a0 con 0x7fe3a815e2a0 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 -- 172.21.15.113:0/2948997172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a815e2a0 msgr2=0x7fe3a806ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2948997172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a815e2a0 0x7fe3a806ba70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe398009840 tx=0x7fe398009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:09.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 -- 172.21.15.113:0/2948997172 shutdown_connections 2024-01-16T13:52:09.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2948997172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a815e2a0 0x7fe3a806ba70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 -- 172.21.15.113:0/2948997172 >> 172.21.15.113:0/2948997172 conn(0x7fe3a80cd570 msgr2=0x7fe3a80cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:09.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 -- 172.21.15.113:0/2948997172 shutdown_connections 2024-01-16T13:52:09.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.183+0000 7fe3b0798700 1 -- 172.21.15.113:0/2948997172 wait complete. 2024-01-16T13:52:09.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.184+0000 7fe3b0798700 1 Processor -- start 2024-01-16T13:52:09.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 -- start start 2024-01-16T13:52:09.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a8073f90 0x7fe3a8144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 0x7fe3a81587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3a815e2a0 0x7fe3a8158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a8159320 con 0x7fe3a80ce090 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a8159600 con 0x7fe3a8073f90 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3b0798700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a81598e0 con 0x7fe3a815e2a0 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3add33700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 0x7fe3a81587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3add33700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 0x7fe3a81587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40342/0 (socket says 172.21.15.113:40342) 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3add33700 1 -- 172.21.15.113:0/2077093404 learned_addr learned my addr 172.21.15.113:0/2077093404 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3ae534700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a8073f90 0x7fe3a8144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.185+0000 7fe3aed35700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3a815e2a0 0x7fe3a8158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3add33700 1 -- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3a815e2a0 msgr2=0x7fe3a8158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3add33700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3a815e2a0 0x7fe3a8158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3add33700 1 -- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a8073f90 msgr2=0x7fe3a8144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3add33700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a8073f90 0x7fe3a8144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3add33700 1 -- 172.21.15.113:0/2077093404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980094a0 con 0x7fe3a80ce090 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3add33700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 0x7fe3a81587b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400c4d0 tx=0x7fe3a400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4011790 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a40118f0 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a8159bc0 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4011ca0 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.186+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a8154ee0 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.187+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a815a0e0 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.187+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe3a401c460 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.188+0000 7fe39effd700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe38c06b350 0x7fe38c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.188+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3a4089830 con 0x7fe3a80ce090 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.188+0000 7fe3ae534700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe38c06b350 0x7fe38c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.188+0000 7fe3ae534700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe38c06b350 0x7fe38c06d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3980090b0 tx=0x7fe398008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.192+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3a815a0e0 con 0x7fe3a80ce090 2024-01-16T13:52:09.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.378+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3a80ce650 con 0x7fe38c06b350 2024-01-16T13:52:09.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.385+0000 7fe39effd700 1 -- 172.21.15.113:0/2077093404 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264716 (secure 0 0 0) 0x7fe398006c60 con 0x7fe38c06b350 2024-01-16T13:52:09.371 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:09.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.388+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe38c06b350 msgr2=0x7fe38c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.388+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe38c06b350 0x7fe38c06d810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3980090b0 tx=0x7fe398008030 comp rx=0 tx=0).stop 2024-01-16T13:52:09.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 msgr2=0x7fe3a81587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 0x7fe3a81587b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400c4d0 tx=0x7fe3a400c7e0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 shutdown_connections 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe38c06b350 0x7fe38c06d810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3a815e2a0 0x7fe3a8158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a8073f90 0x7fe3a8144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 --2- 172.21.15.113:0/2077093404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3a80ce090 0x7fe3a81587b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 >> 172.21.15.113:0/2077093404 conn(0x7fe3a80cd570 msgr2=0x7fe3a80cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 shutdown_connections 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.389+0000 7fe3b0798700 1 -- 172.21.15.113:0/2077093404 wait complete. 2024-01-16T13:52:09.374 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:52:09.383 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-01-16T13:52:08.402241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.516646"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383364+0000","last_change":"2024-01-16T13:52:04.933545+0000","last_active":"2024-01-16T13:52:06.383364+0000","last_peered":"2024-01-16T13:52:06.383364+0000","last_clean":"2024-01-16T13:52:06.383364+0000","last_became_active":"2024-01-16T13:52:04.932314+0000","last_became_peered":"2024-01-16T13:52:04.932314+0000","last_unstale":"2024-01-16T13:52:06.383364+0000","last_undegraded":"2024-01-16T13:52:06.383364+0000","last_fullsized":"2024-01-16T13:52:06.383364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.917385+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.916958+0000","last_became_peered":"2024-01-16T13:52:04.916958+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382533+0000","last_change":"2024-01-16T13:52:04.917514+0000","last_active":"2024-01-16T13:52:06.382533+0000","last_peered":"2024-01-16T13:52:06.382533+0000","last_clean":"2024-01-16T13:52:06.382533+0000","last_became_active":"2024-01-16T13:52:04.916822+0000","last_became_peered":"2024-01-16T13:52:04.916822+0000","last_unstale":"2024-01-16T13:52:06.382533+0000","last_undegraded":"2024-01-16T13:52:06.382533+0000","last_fullsized":"2024-01-16T13:52:06.382533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:05.397603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904528+0000","last_change":"2024-01-16T13:52:04.926700+0000","last_active":"2024-01-16T13:52:05.904528+0000","last_peered":"2024-01-16T13:52:05.904528+0000","last_clean":"2024-01-16T13:52:05.904528+0000","last_became_active":"2024-01-16T13:52:04.923728+0000","last_became_peered":"2024-01-16T13:52:04.923728+0000","last_unstale":"2024-01-16T13:52:05.904528+0000","last_undegraded":"2024-01-16T13:52:05.904528+0000","last_fullsized":"2024-01-16T13:52:05.904528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382807+0000","last_change":"2024-01-16T13:52:04.931697+0000","last_active":"2024-01-16T13:52:06.382807+0000","last_peered":"2024-01-16T13:52:06.382807+0000","last_clean":"2024-01-16T13:52:06.382807+0000","last_became_active":"2024-01-16T13:52:04.931415+0000","last_became_peered":"2024-01-16T13:52:04.931415+0000","last_unstale":"2024-01-16T13:52:06.382807+0000","last_undegraded":"2024-01-16T13:52:06.382807+0000","last_fullsized":"2024-01-16T13:52:06.382807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904991+0000","last_change":"2024-01-16T13:52:04.925527+0000","last_active":"2024-01-16T13:52:05.904991+0000","last_peered":"2024-01-16T13:52:05.904991+0000","last_clean":"2024-01-16T13:52:05.904991+0000","last_became_active":"2024-01-16T13:52:04.923026+0000","last_became_peered":"2024-01-16T13:52:04.923026+0000","last_unstale":"2024-01-16T13:52:05.904991+0000","last_undegraded":"2024-01-16T13:52:05.904991+0000","last_fullsized":"2024-01-16T13:52:05.904991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382610+0000","last_change":"2024-01-16T13:52:04.930095+0000","last_active":"2024-01-16T13:52:06.382610+0000","last_peered":"2024-01-16T13:52:06.382610+0000","last_clean":"2024-01-16T13:52:06.382610+0000","last_became_active":"2024-01-16T13:52:04.926632+0000","last_became_peered":"2024-01-16T13:52:04.926632+0000","last_unstale":"2024-01-16T13:52:06.382610+0000","last_undegraded":"2024-01-16T13:52:06.382610+0000","last_fullsized":"2024-01-16T13:52:06.382610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382842+0000","last_change":"2024-01-16T13:52:04.927129+0000","last_active":"2024-01-16T13:52:06.382842+0000","last_peered":"2024-01-16T13:52:06.382842+0000","last_clean":"2024-01-16T13:52:06.382842+0000","last_became_active":"2024-01-16T13:52:04.926541+0000","last_became_peered":"2024-01-16T13:52:04.926541+0000","last_unstale":"2024-01-16T13:52:06.382842+0000","last_undegraded":"2024-01-16T13:52:06.382842+0000","last_fullsized":"2024-01-16T13:52:06.382842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383203+0000","last_change":"2024-01-16T13:52:04.935989+0000","last_active":"2024-01-16T13:52:06.383203+0000","last_peered":"2024-01-16T13:52:06.383203+0000","last_clean":"2024-01-16T13:52:06.383203+0000","last_became_active":"2024-01-16T13:52:04.935750+0000","last_became_peered":"2024-01-16T13:52:04.935750+0000","last_unstale":"2024-01-16T13:52:06.383203+0000","last_undegraded":"2024-01-16T13:52:06.383203+0000","last_fullsized":"2024-01-16T13:52:06.383203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904269+0000","last_change":"2024-01-16T13:52:04.941805+0000","last_active":"2024-01-16T13:52:05.904269+0000","last_peered":"2024-01-16T13:52:05.904269+0000","last_clean":"2024-01-16T13:52:05.904269+0000","last_became_active":"2024-01-16T13:52:04.941552+0000","last_became_peered":"2024-01-16T13:52:04.941552+0000","last_unstale":"2024-01-16T13:52:05.904269+0000","last_undegraded":"2024-01-16T13:52:05.904269+0000","last_fullsized":"2024-01-16T13:52:05.904269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904181+0000","last_change":"2024-01-16T13:52:04.941357+0000","last_active":"2024-01-16T13:52:05.904181+0000","last_peered":"2024-01-16T13:52:05.904181+0000","last_clean":"2024-01-16T13:52:05.904181+0000","last_became_active":"2024-01-16T13:52:04.940946+0000","last_became_peered":"2024-01-16T13:52:04.940946+0000","last_unstale":"2024-01-16T13:52:05.904181+0000","last_undegraded":"2024-01-16T13:52:05.904181+0000","last_fullsized":"2024-01-16T13:52:05.904181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383260+0000","last_change":"2024-01-16T13:52:04.923980+0000","last_active":"2024-01-16T13:52:06.383260+0000","last_peered":"2024-01-16T13:52:06.383260+0000","last_clean":"2024-01-16T13:52:06.383260+0000","last_became_active":"2024-01-16T13:52:04.923398+0000","last_became_peered":"2024-01-16T13:52:04.923398+0000","last_unstale":"2024-01-16T13:52:06.383260+0000","last_undegraded":"2024-01-16T13:52:06.383260+0000","last_fullsized":"2024-01-16T13:52:06.383260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383179+0000","last_change":"2024-01-16T13:52:04.922440+0000","last_active":"2024-01-16T13:52:06.383179+0000","last_peered":"2024-01-16T13:52:06.383179+0000","last_clean":"2024-01-16T13:52:06.383179+0000","last_became_active":"2024-01-16T13:52:04.922170+0000","last_became_peered":"2024-01-16T13:52:04.922170+0000","last_unstale":"2024-01-16T13:52:06.383179+0000","last_undegraded":"2024-01-16T13:52:06.383179+0000","last_fullsized":"2024-01-16T13:52:06.383179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382935+0000","last_change":"2024-01-16T13:52:04.918126+0000","last_active":"2024-01-16T13:52:06.382935+0000","last_peered":"2024-01-16T13:52:06.382935+0000","last_clean":"2024-01-16T13:52:06.382935+0000","last_became_active":"2024-01-16T13:52:04.917683+0000","last_became_peered":"2024-01-16T13:52:04.917683+0000","last_unstale":"2024-01-16T13:52:06.382935+0000","last_undegraded":"2024-01-16T13:52:06.382935+0000","last_fullsized":"2024-01-16T13:52:06.382935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904662+0000","last_change":"2024-01-16T13:52:04.919145+0000","last_active":"2024-01-16T13:52:05.904662+0000","last_peered":"2024-01-16T13:52:05.904662+0000","last_clean":"2024-01-16T13:52:05.904662+0000","last_became_active":"2024-01-16T13:52:04.918897+0000","last_became_peered":"2024-01-16T13:52:04.918897+0000","last_unstale":"2024-01-16T13:52:05.904662+0000","last_undegraded":"2024-01-16T13:52:05.904662+0000","last_fullsized":"2024-01-16T13:52:05.904662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.923697+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.923053+0000","last_became_peered":"2024-01-16T13:52:04.923053+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904261+0000","last_change":"2024-01-16T13:52:04.927671+0000","last_active":"2024-01-16T13:52:05.904261+0000","last_peered":"2024-01-16T13:52:05.904261+0000","last_clean":"2024-01-16T13:52:05.904261+0000","last_became_active":"2024-01-16T13:52:04.927402+0000","last_became_peered":"2024-01-16T13:52:04.927402+0000","last_unstale":"2024-01-16T13:52:05.904261+0000","last_undegraded":"2024-01-16T13:52:05.904261+0000","last_fullsized":"2024-01-16T13:52:05.904261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382408+0000","last_change":"2024-01-16T13:52:04.924972+0000","last_active":"2024-01-16T13:52:06.382408+0000","last_peered":"2024-01-16T13:52:06.382408+0000","last_clean":"2024-01-16T13:52:06.382408+0000","last_became_active":"2024-01-16T13:52:04.924756+0000","last_became_peered":"2024-01-16T13:52:04.924756+0000","last_unstale":"2024-01-16T13:52:06.382408+0000","last_undegraded":"2024-01-16T13:52:06.382408+0000","last_fullsized":"2024-01-16T13:52:06.382408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383026+0000","last_change":"2024-01-16T13:52:04.932352+0000","last_active":"2024-01-16T13:52:06.383026+0000","last_peered":"2024-01-16T13:52:06.383026+0000","last_clean":"2024-01-16T13:52:06.383026+0000","last_became_active":"2024-01-16T13:52:04.926615+0000","last_became_peered":"2024-01-16T13:52:04.926615+0000","last_unstale":"2024-01-16T13:52:06.383026+0000","last_undegraded":"2024-01-16T13:52:06.383026+0000","last_fullsized":"2024-01-16T13:52:06.383026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382783+0000","last_change":"2024-01-16T13:52:03.906795+0000","last_active":"2024-01-16T13:52:06.382783+0000","last_peered":"2024-01-16T13:52:06.382783+0000","last_clean":"2024-01-16T13:52:06.382783+0000","last_became_active":"2024-01-16T13:52:03.906413+0000","last_became_peered":"2024-01-16T13:52:03.906413+0000","last_unstale":"2024-01-16T13:52:06.382783+0000","last_undegraded":"2024-01-16T13:52:06.382783+0000","last_fullsized":"2024-01-16T13:52:06.382783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383205+0000","last_change":"2024-01-16T13:52:04.922947+0000","last_active":"2024-01-16T13:52:06.383205+0000","last_peered":"2024-01-16T13:52:06.383205+0000","last_clean":"2024-01-16T13:52:06.383205+0000","last_became_active":"2024-01-16T13:52:04.922641+0000","last_became_peered":"2024-01-16T13:52:04.922641+0000","last_unstale":"2024-01-16T13:52:06.383205+0000","last_undegraded":"2024-01-16T13:52:06.383205+0000","last_fullsized":"2024-01-16T13:52:06.383205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383275+0000","last_change":"2024-01-16T13:52:03.911364+0000","last_active":"2024-01-16T13:52:06.383275+0000","last_peered":"2024-01-16T13:52:06.383275+0000","last_clean":"2024-01-16T13:52:06.383275+0000","last_became_active":"2024-01-16T13:52:03.911096+0000","last_became_peered":"2024-01-16T13:52:03.911096+0000","last_unstale":"2024-01-16T13:52:06.383275+0000","last_undegraded":"2024-01-16T13:52:06.383275+0000","last_fullsized":"2024-01-16T13:52:06.383275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382855+0000","last_change":"2024-01-16T13:52:04.941583+0000","last_active":"2024-01-16T13:52:06.382855+0000","last_peered":"2024-01-16T13:52:06.382855+0000","last_clean":"2024-01-16T13:52:06.382855+0000","last_became_active":"2024-01-16T13:52:04.941270+0000","last_became_peered":"2024-01-16T13:52:04.941270+0000","last_unstale":"2024-01-16T13:52:06.382855+0000","last_undegraded":"2024-01-16T13:52:06.382855+0000","last_fullsized":"2024-01-16T13:52:06.382855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383462+0000","last_change":"2024-01-16T13:52:03.908013+0000","last_active":"2024-01-16T13:52:06.383462+0000","last_peered":"2024-01-16T13:52:06.383462+0000","last_clean":"2024-01-16T13:52:06.383462+0000","last_became_active":"2024-01-16T13:52:03.907716+0000","last_became_peered":"2024-01-16T13:52:03.907716+0000","last_unstale":"2024-01-16T13:52:06.383462+0000","last_undegraded":"2024-01-16T13:52:06.383462+0000","last_fullsized":"2024-01-16T13:52:06.383462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904017+0000","last_change":"2024-01-16T13:52:04.920391+0000","last_active":"2024-01-16T13:52:05.904017+0000","last_peered":"2024-01-16T13:52:05.904017+0000","last_clean":"2024-01-16T13:52:05.904017+0000","last_became_active":"2024-01-16T13:52:04.920118+0000","last_became_peered":"2024-01-16T13:52:04.920118+0000","last_unstale":"2024-01-16T13:52:05.904017+0000","last_undegraded":"2024-01-16T13:52:05.904017+0000","last_fullsized":"2024-01-16T13:52:05.904017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382906+0000","last_change":"2024-01-16T13:52:03.905310+0000","last_active":"2024-01-16T13:52:06.382906+0000","last_peered":"2024-01-16T13:52:06.382906+0000","last_clean":"2024-01-16T13:52:06.382906+0000","last_became_active":"2024-01-16T13:52:03.905031+0000","last_became_peered":"2024-01-16T13:52:03.905031+0000","last_unstale":"2024-01-16T13:52:06.382906+0000","last_undegraded":"2024-01-16T13:52:06.382906+0000","last_fullsized":"2024-01-16T13:52:06.382906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904705+0000","last_change":"2024-01-16T13:52:04.912084+0000","last_active":"2024-01-16T13:52:05.904705+0000","last_peered":"2024-01-16T13:52:05.904705+0000","last_clean":"2024-01-16T13:52:05.904705+0000","last_became_active":"2024-01-16T13:52:04.911792+0000","last_became_peered":"2024-01-16T13:52:04.911792+0000","last_unstale":"2024-01-16T13:52:05.904705+0000","last_undegraded":"2024-01-16T13:52:05.904705+0000","last_fullsized":"2024-01-16T13:52:05.904705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383224+0000","last_change":"2024-01-16T13:52:03.912985+0000","last_active":"2024-01-16T13:52:06.383224+0000","last_peered":"2024-01-16T13:52:06.383224+0000","last_clean":"2024-01-16T13:52:06.383224+0000","last_became_active":"2024-01-16T13:52:03.912758+0000","last_became_peered":"2024-01-16T13:52:03.912758+0000","last_unstale":"2024-01-16T13:52:06.383224+0000","last_undegraded":"2024-01-16T13:52:06.383224+0000","last_fullsized":"2024-01-16T13:52:06.383224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382525+0000","last_change":"2024-01-16T13:52:04.917347+0000","last_active":"2024-01-16T13:52:06.382525+0000","last_peered":"2024-01-16T13:52:06.382525+0000","last_clean":"2024-01-16T13:52:06.382525+0000","last_became_active":"2024-01-16T13:52:04.916918+0000","last_became_peered":"2024-01-16T13:52:04.916918+0000","last_unstale":"2024-01-16T13:52:06.382525+0000","last_undegraded":"2024-01-16T13:52:06.382525+0000","last_fullsized":"2024-01-16T13:52:06.382525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:51.832998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903890+0000","last_change":"2024-01-16T13:52:03.903840+0000","last_active":"2024-01-16T13:52:05.903890+0000","last_peered":"2024-01-16T13:52:05.903890+0000","last_clean":"2024-01-16T13:52:05.903890+0000","last_became_active":"2024-01-16T13:52:03.903524+0000","last_became_peered":"2024-01-16T13:52:03.903524+0000","last_unstale":"2024-01-16T13:52:05.903890+0000","last_undegraded":"2024-01-16T13:52:05.903890+0000","last_fullsized":"2024-01-16T13:52:05.903890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383068+0000","last_change":"2024-01-16T13:52:04.924992+0000","last_active":"2024-01-16T13:52:06.383068+0000","last_peered":"2024-01-16T13:52:06.383068+0000","last_clean":"2024-01-16T13:52:06.383068+0000","last_became_active":"2024-01-16T13:52:04.924689+0000","last_became_peered":"2024-01-16T13:52:04.924689+0000","last_unstale":"2024-01-16T13:52:06.383068+0000","last_undegraded":"2024-01-16T13:52:06.383068+0000","last_fullsized":"2024-01-16T13:52:06.383068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383042+0000","last_change":"2024-01-16T13:52:03.912658+0000","last_active":"2024-01-16T13:52:06.383042+0000","last_peered":"2024-01-16T13:52:06.383042+0000","last_clean":"2024-01-16T13:52:06.383042+0000","last_became_active":"2024-01-16T13:52:03.912382+0000","last_became_peered":"2024-01-16T13:52:03.912382+0000","last_unstale":"2024-01-16T13:52:06.383042+0000","last_undegraded":"2024-01-16T13:52:06.383042+0000","last_fullsized":"2024-01-16T13:52:06.383042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383509+0000","last_change":"2024-01-16T13:52:04.920332+0000","last_active":"2024-01-16T13:52:06.383509+0000","last_peered":"2024-01-16T13:52:06.383509+0000","last_clean":"2024-01-16T13:52:06.383509+0000","last_became_active":"2024-01-16T13:52:04.920061+0000","last_became_peered":"2024-01-16T13:52:04.920061+0000","last_unstale":"2024-01-16T13:52:06.383509+0000","last_undegraded":"2024-01-16T13:52:06.383509+0000","last_fullsized":"2024-01-16T13:52:06.383509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904985+0000","last_change":"2024-01-16T13:52:03.897543+0000","last_active":"2024-01-16T13:52:05.904985+0000","last_peered":"2024-01-16T13:52:05.904985+0000","last_clean":"2024-01-16T13:52:05.904985+0000","last_became_active":"2024-01-16T13:52:03.897059+0000","last_became_peered":"2024-01-16T13:52:03.897059+0000","last_unstale":"2024-01-16T13:52:05.904985+0000","last_undegraded":"2024-01-16T13:52:05.904985+0000","last_fullsized":"2024-01-16T13:52:05.904985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904732+0000","last_change":"2024-01-16T13:52:04.906828+0000","last_active":"2024-01-16T13:52:05.904732+0000","last_peered":"2024-01-16T13:52:05.904732+0000","last_clean":"2024-01-16T13:52:05.904732+0000","last_became_active":"2024-01-16T13:52:04.906549+0000","last_became_peered":"2024-01-16T13:52:04.906549+0000","last_unstale":"2024-01-16T13:52:05.904732+0000","last_undegraded":"2024-01-16T13:52:05.904732+0000","last_fullsized":"2024-01-16T13:52:05.904732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904051+0000","last_change":"2024-01-16T13:52:03.901711+0000","last_active":"2024-01-16T13:52:05.904051+0000","last_peered":"2024-01-16T13:52:05.904051+0000","last_clean":"2024-01-16T13:52:05.904051+0000","last_became_active":"2024-01-16T13:52:03.901420+0000","last_became_peered":"2024-01-16T13:52:03.901420+0000","last_unstale":"2024-01-16T13:52:05.904051+0000","last_undegraded":"2024-01-16T13:52:05.904051+0000","last_fullsized":"2024-01-16T13:52:05.904051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904698+0000","last_change":"2024-01-16T13:52:04.935326+0000","last_active":"2024-01-16T13:52:05.904698+0000","last_peered":"2024-01-16T13:52:05.904698+0000","last_clean":"2024-01-16T13:52:05.904698+0000","last_became_active":"2024-01-16T13:52:04.934628+0000","last_became_peered":"2024-01-16T13:52:04.934628+0000","last_unstale":"2024-01-16T13:52:05.904698+0000","last_undegraded":"2024-01-16T13:52:05.904698+0000","last_fullsized":"2024-01-16T13:52:05.904698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904576+0000","last_change":"2024-01-16T13:52:03.902616+0000","last_active":"2024-01-16T13:52:05.904576+0000","last_peered":"2024-01-16T13:52:05.904576+0000","last_clean":"2024-01-16T13:52:05.904576+0000","last_became_active":"2024-01-16T13:52:03.902333+0000","last_became_peered":"2024-01-16T13:52:03.902333+0000","last_unstale":"2024-01-16T13:52:05.904576+0000","last_undegraded":"2024-01-16T13:52:05.904576+0000","last_fullsized":"2024-01-16T13:52:05.904576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383250+0000","last_change":"2024-01-16T13:52:04.936172+0000","last_active":"2024-01-16T13:52:06.383250+0000","last_peered":"2024-01-16T13:52:06.383250+0000","last_clean":"2024-01-16T13:52:06.383250+0000","last_became_active":"2024-01-16T13:52:04.935881+0000","last_became_peered":"2024-01-16T13:52:04.935881+0000","last_unstale":"2024-01-16T13:52:06.383250+0000","last_undegraded":"2024-01-16T13:52:06.383250+0000","last_fullsized":"2024-01-16T13:52:06.383250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904808+0000","last_change":"2024-01-16T13:52:03.896280+0000","last_active":"2024-01-16T13:52:05.904808+0000","last_peered":"2024-01-16T13:52:05.904808+0000","last_clean":"2024-01-16T13:52:05.904808+0000","last_became_active":"2024-01-16T13:52:03.896001+0000","last_became_peered":"2024-01-16T13:52:03.896001+0000","last_unstale":"2024-01-16T13:52:05.904808+0000","last_undegraded":"2024-01-16T13:52:05.904808+0000","last_fullsized":"2024-01-16T13:52:05.904808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383116+0000","last_change":"2024-01-16T13:52:04.927368+0000","last_active":"2024-01-16T13:52:06.383116+0000","last_peered":"2024-01-16T13:52:06.383116+0000","last_clean":"2024-01-16T13:52:06.383116+0000","last_became_active":"2024-01-16T13:52:04.927100+0000","last_became_peered":"2024-01-16T13:52:04.927100+0000","last_unstale":"2024-01-16T13:52:06.383116+0000","last_undegraded":"2024-01-16T13:52:06.383116+0000","last_fullsized":"2024-01-16T13:52:06.383116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904369+0000","last_change":"2024-01-16T13:52:03.910360+0000","last_active":"2024-01-16T13:52:05.904369+0000","last_peered":"2024-01-16T13:52:05.904369+0000","last_clean":"2024-01-16T13:52:05.904369+0000","last_became_active":"2024-01-16T13:52:03.910098+0000","last_became_peered":"2024-01-16T13:52:03.910098+0000","last_unstale":"2024-01-16T13:52:05.904369+0000","last_undegraded":"2024-01-16T13:52:05.904369+0000","last_fullsized":"2024-01-16T13:52:05.904369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383195+0000","last_change":"2024-01-16T13:52:04.932597+0000","last_active":"2024-01-16T13:52:06.383195+0000","last_peered":"2024-01-16T13:52:06.383195+0000","last_clean":"2024-01-16T13:52:06.383195+0000","last_became_active":"2024-01-16T13:52:04.930511+0000","last_became_peered":"2024-01-16T13:52:04.930511+0000","last_unstale":"2024-01-16T13:52:06.383195+0000","last_undegraded":"2024-01-16T13:52:06.383195+0000","last_fullsized":"2024-01-16T13:52:06.383195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382997+0000","last_change":"2024-01-16T13:52:03.906336+0000","last_active":"2024-01-16T13:52:06.382997+0000","last_peered":"2024-01-16T13:52:06.382997+0000","last_clean":"2024-01-16T13:52:06.382997+0000","last_became_active":"2024-01-16T13:52:03.905656+0000","last_became_peered":"2024-01-16T13:52:03.905656+0000","last_unstale":"2024-01-16T13:52:06.382997+0000","last_undegraded":"2024-01-16T13:52:06.382997+0000","last_fullsized":"2024-01-16T13:52:06.382997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382700+0000","last_change":"2024-01-16T13:52:04.927208+0000","last_active":"2024-01-16T13:52:06.382700+0000","last_peered":"2024-01-16T13:52:06.382700+0000","last_clean":"2024-01-16T13:52:06.382700+0000","last_became_active":"2024-01-16T13:52:04.926931+0000","last_became_peered":"2024-01-16T13:52:04.926931+0000","last_unstale":"2024-01-16T13:52:06.382700+0000","last_undegraded":"2024-01-16T13:52:06.382700+0000","last_fullsized":"2024-01-16T13:52:06.382700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905228+0000","last_change":"2024-01-16T13:52:03.911290+0000","last_active":"2024-01-16T13:52:05.905228+0000","last_peered":"2024-01-16T13:52:05.905228+0000","last_clean":"2024-01-16T13:52:05.905228+0000","last_became_active":"2024-01-16T13:52:03.911029+0000","last_became_peered":"2024-01-16T13:52:03.911029+0000","last_unstale":"2024-01-16T13:52:05.905228+0000","last_undegraded":"2024-01-16T13:52:05.905228+0000","last_fullsized":"2024-01-16T13:52:05.905228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:40.789717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383244+0000","last_change":"2024-01-16T13:52:04.932349+0000","last_active":"2024-01-16T13:52:06.383244+0000","last_peered":"2024-01-16T13:52:06.383244+0000","last_clean":"2024-01-16T13:52:06.383244+0000","last_became_active":"2024-01-16T13:52:04.931854+0000","last_became_peered":"2024-01-16T13:52:04.931854+0000","last_unstale":"2024-01-16T13:52:06.383244+0000","last_undegraded":"2024-01-16T13:52:06.383244+0000","last_fullsized":"2024-01-16T13:52:06.383244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903653+0000","last_change":"2024-01-16T13:52:03.903815+0000","last_active":"2024-01-16T13:52:05.903653+0000","last_peered":"2024-01-16T13:52:05.903653+0000","last_clean":"2024-01-16T13:52:05.903653+0000","last_became_active":"2024-01-16T13:52:03.903487+0000","last_became_peered":"2024-01-16T13:52:03.903487+0000","last_unstale":"2024-01-16T13:52:05.903653+0000","last_undegraded":"2024-01-16T13:52:05.903653+0000","last_fullsized":"2024-01-16T13:52:05.903653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:49.732984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903921+0000","last_change":"2024-01-16T13:52:04.922443+0000","last_active":"2024-01-16T13:52:05.903921+0000","last_peered":"2024-01-16T13:52:05.903921+0000","last_clean":"2024-01-16T13:52:05.903921+0000","last_became_active":"2024-01-16T13:52:04.922131+0000","last_became_peered":"2024-01-16T13:52:04.922131+0000","last_unstale":"2024-01-16T13:52:05.903921+0000","last_undegraded":"2024-01-16T13:52:05.903921+0000","last_fullsized":"2024-01-16T13:52:05.903921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382223+0000","last_change":"2024-01-16T13:52:03.910304+0000","last_active":"2024-01-16T13:52:06.382223+0000","last_peered":"2024-01-16T13:52:06.382223+0000","last_clean":"2024-01-16T13:52:06.382223+0000","last_became_active":"2024-01-16T13:52:03.910009+0000","last_became_peered":"2024-01-16T13:52:03.910009+0000","last_unstale":"2024-01-16T13:52:06.382223+0000","last_undegraded":"2024-01-16T13:52:06.382223+0000","last_fullsized":"2024-01-16T13:52:06.382223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382979+0000","last_change":"2024-01-16T13:52:04.930880+0000","last_active":"2024-01-16T13:52:06.382979+0000","last_peered":"2024-01-16T13:52:06.382979+0000","last_clean":"2024-01-16T13:52:06.382979+0000","last_became_active":"2024-01-16T13:52:04.930553+0000","last_became_peered":"2024-01-16T13:52:04.930553+0000","last_unstale":"2024-01-16T13:52:06.382979+0000","last_undegraded":"2024-01-16T13:52:06.382979+0000","last_fullsized":"2024-01-16T13:52:06.382979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382946+0000","last_change":"2024-01-16T13:52:03.906573+0000","last_active":"2024-01-16T13:52:06.382946+0000","last_peered":"2024-01-16T13:52:06.382946+0000","last_clean":"2024-01-16T13:52:06.382946+0000","last_became_active":"2024-01-16T13:52:03.906257+0000","last_became_peered":"2024-01-16T13:52:03.906257+0000","last_unstale":"2024-01-16T13:52:06.382946+0000","last_undegraded":"2024-01-16T13:52:06.382946+0000","last_fullsized":"2024-01-16T13:52:06.382946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:14:03.506561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382905+0000","last_change":"2024-01-16T13:52:04.931581+0000","last_active":"2024-01-16T13:52:06.382905+0000","last_peered":"2024-01-16T13:52:06.382905+0000","last_clean":"2024-01-16T13:52:06.382905+0000","last_became_active":"2024-01-16T13:52:04.931306+0000","last_became_peered":"2024-01-16T13:52:04.931306+0000","last_unstale":"2024-01-16T13:52:06.382905+0000","last_undegraded":"2024-01-16T13:52:06.382905+0000","last_fullsized":"2024-01-16T13:52:06.382905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904968+0000","last_change":"2024-01-16T13:52:03.903873+0000","last_active":"2024-01-16T13:52:05.904968+0000","last_peered":"2024-01-16T13:52:05.904968+0000","last_clean":"2024-01-16T13:52:05.904968+0000","last_became_active":"2024-01-16T13:52:03.903582+0000","last_became_peered":"2024-01-16T13:52:03.903582+0000","last_unstale":"2024-01-16T13:52:05.904968+0000","last_undegraded":"2024-01-16T13:52:05.904968+0000","last_fullsized":"2024-01-16T13:52:05.904968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904494+0000","last_change":"2024-01-16T13:52:04.926087+0000","last_active":"2024-01-16T13:52:05.904494+0000","last_peered":"2024-01-16T13:52:05.904494+0000","last_clean":"2024-01-16T13:52:05.904494+0000","last_became_active":"2024-01-16T13:52:04.925736+0000","last_became_peered":"2024-01-16T13:52:04.925736+0000","last_unstale":"2024-01-16T13:52:05.904494+0000","last_undegraded":"2024-01-16T13:52:05.904494+0000","last_fullsized":"2024-01-16T13:52:05.904494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905152+0000","last_change":"2024-01-16T13:52:03.901400+0000","last_active":"2024-01-16T13:52:05.905152+0000","last_peered":"2024-01-16T13:52:05.905152+0000","last_clean":"2024-01-16T13:52:05.905152+0000","last_became_active":"2024-01-16T13:52:03.901087+0000","last_became_peered":"2024-01-16T13:52:03.901087+0000","last_unstale":"2024-01-16T13:52:05.905152+0000","last_undegraded":"2024-01-16T13:52:05.905152+0000","last_fullsized":"2024-01-16T13:52:05.905152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:43.211649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382982+0000","last_change":"2024-01-16T13:52:04.917916+0000","last_active":"2024-01-16T13:52:06.382982+0000","last_peered":"2024-01-16T13:52:06.382982+0000","last_clean":"2024-01-16T13:52:06.382982+0000","last_became_active":"2024-01-16T13:52:04.917649+0000","last_became_peered":"2024-01-16T13:52:04.917649+0000","last_unstale":"2024-01-16T13:52:06.382982+0000","last_undegraded":"2024-01-16T13:52:06.382982+0000","last_fullsized":"2024-01-16T13:52:06.382982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:37.904615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904621+0000","last_change":"2024-01-16T13:52:04.916017+0000","last_active":"2024-01-16T13:52:05.904621+0000","last_peered":"2024-01-16T13:52:05.904621+0000","last_clean":"2024-01-16T13:52:05.904621+0000","last_became_active":"2024-01-16T13:52:04.915765+0000","last_became_peered":"2024-01-16T13:52:04.915765+0000","last_unstale":"2024-01-16T13:52:05.904621+0000","last_undegraded":"2024-01-16T13:52:05.904621+0000","last_fullsized":"2024-01-16T13:52:05.904621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382686+0000","last_change":"2024-01-16T13:52:03.910443+0000","last_active":"2024-01-16T13:52:06.382686+0000","last_peered":"2024-01-16T13:52:06.382686+0000","last_clean":"2024-01-16T13:52:06.382686+0000","last_became_active":"2024-01-16T13:52:03.910025+0000","last_became_peered":"2024-01-16T13:52:03.910025+0000","last_unstale":"2024-01-16T13:52:06.382686+0000","last_undegraded":"2024-01-16T13:52:06.382686+0000","last_fullsized":"2024-01-16T13:52:06.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:22.047669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383379+0000","last_change":"2024-01-16T13:52:04.927591+0000","last_active":"2024-01-16T13:52:06.383379+0000","last_peered":"2024-01-16T13:52:06.383379+0000","last_clean":"2024-01-16T13:52:06.383379+0000","last_became_active":"2024-01-16T13:52:04.927020+0000","last_became_peered":"2024-01-16T13:52:04.927020+0000","last_unstale":"2024-01-16T13:52:06.383379+0000","last_undegraded":"2024-01-16T13:52:06.383379+0000","last_fullsized":"2024-01-16T13:52:06.383379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903799+0000","last_change":"2024-01-16T13:52:03.906447+0000","last_active":"2024-01-16T13:52:05.903799+0000","last_peered":"2024-01-16T13:52:05.903799+0000","last_clean":"2024-01-16T13:52:05.903799+0000","last_became_active":"2024-01-16T13:52:03.906082+0000","last_became_peered":"2024-01-16T13:52:03.906082+0000","last_unstale":"2024-01-16T13:52:05.903799+0000","last_undegraded":"2024-01-16T13:52:05.903799+0000","last_fullsized":"2024-01-16T13:52:05.903799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T13:52:04.913338+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:04.913338+0000","last_peered":"2024-01-16T13:52:04.913338+0000","last_clean":"2024-01-16T13:52:04.913338+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:04.913338+0000","last_undegraded":"2024-01-16T13:52:04.913338+0000","last_fullsized":"2024-01-16T13:52:04.913338+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382590+0000","last_change":"2024-01-16T13:52:04.923544+0000","last_active":"2024-01-16T13:52:06.382590+0000","last_peered":"2024-01-16T13:52:06.382590+0000","last_clean":"2024-01-16T13:52:06.382590+0000","last_became_active":"2024-01-16T13:52:04.923283+0000","last_became_peered":"2024-01-16T13:52:04.923283+0000","last_unstale":"2024-01-16T13:52:06.382590+0000","last_undegraded":"2024-01-16T13:52:06.382590+0000","last_fullsized":"2024-01-16T13:52:06.382590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382524+0000","last_change":"2024-01-16T13:52:03.910495+0000","last_active":"2024-01-16T13:52:06.382524+0000","last_peered":"2024-01-16T13:52:06.382524+0000","last_clean":"2024-01-16T13:52:06.382524+0000","last_became_active":"2024-01-16T13:52:03.910162+0000","last_became_peered":"2024-01-16T13:52:03.910162+0000","last_unstale":"2024-01-16T13:52:06.382524+0000","last_undegraded":"2024-01-16T13:52:06.382524+0000","last_fullsized":"2024-01-16T13:52:06.382524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:44.922338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:03.910098+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:03.909843+0000","last_became_peered":"2024-01-16T13:52:03.909843+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904899+0000","last_change":"2024-01-16T13:52:04.915509+0000","last_active":"2024-01-16T13:52:05.904899+0000","last_peered":"2024-01-16T13:52:05.904899+0000","last_clean":"2024-01-16T13:52:05.904899+0000","last_became_active":"2024-01-16T13:52:04.915240+0000","last_became_peered":"2024-01-16T13:52:04.915240+0000","last_unstale":"2024-01-16T13:52:05.904899+0000","last_undegraded":"2024-01-16T13:52:05.904899+0000","last_fullsized":"2024-01-16T13:52:05.904899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904829+0000","last_change":"2024-01-16T13:52:03.903483+0000","last_active":"2024-01-16T13:52:05.904829+0000","last_peered":"2024-01-16T13:52:05.904829+0000","last_clean":"2024-01-16T13:52:05.904829+0000","last_became_active":"2024-01-16T13:52:03.903147+0000","last_became_peered":"2024-01-16T13:52:03.903147+0000","last_unstale":"2024-01-16T13:52:05.904829+0000","last_undegraded":"2024-01-16T13:52:05.904829+0000","last_fullsized":"2024-01-16T13:52:05.904829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383351+0000","last_change":"2024-01-16T13:52:04.927887+0000","last_active":"2024-01-16T13:52:06.383351+0000","last_peered":"2024-01-16T13:52:06.383351+0000","last_clean":"2024-01-16T13:52:06.383351+0000","last_became_active":"2024-01-16T13:52:04.927626+0000","last_became_peered":"2024-01-16T13:52:04.927626+0000","last_unstale":"2024-01-16T13:52:06.383351+0000","last_undegraded":"2024-01-16T13:52:06.383351+0000","last_fullsized":"2024-01-16T13:52:06.383351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904612+0000","last_change":"2024-01-16T13:52:03.903609+0000","last_active":"2024-01-16T13:52:05.904612+0000","last_peered":"2024-01-16T13:52:05.904612+0000","last_clean":"2024-01-16T13:52:05.904612+0000","last_became_active":"2024-01-16T13:52:03.903258+0000","last_became_peered":"2024-01-16T13:52:03.903258+0000","last_unstale":"2024-01-16T13:52:05.904612+0000","last_undegraded":"2024-01-16T13:52:05.904612+0000","last_fullsized":"2024-01-16T13:52:05.904612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383138+0000","last_change":"2024-01-16T13:52:04.918025+0000","last_active":"2024-01-16T13:52:06.383138+0000","last_peered":"2024-01-16T13:52:06.383138+0000","last_clean":"2024-01-16T13:52:06.383138+0000","last_became_active":"2024-01-16T13:52:04.917716+0000","last_became_peered":"2024-01-16T13:52:04.917716+0000","last_unstale":"2024-01-16T13:52:06.383138+0000","last_undegraded":"2024-01-16T13:52:06.383138+0000","last_fullsized":"2024-01-16T13:52:06.383138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382341+0000","last_change":"2024-01-16T13:52:03.909845+0000","last_active":"2024-01-16T13:52:06.382341+0000","last_peered":"2024-01-16T13:52:06.382341+0000","last_clean":"2024-01-16T13:52:06.382341+0000","last_became_active":"2024-01-16T13:52:03.909573+0000","last_became_peered":"2024-01-16T13:52:03.909573+0000","last_unstale":"2024-01-16T13:52:06.382341+0000","last_undegraded":"2024-01-16T13:52:06.382341+0000","last_fullsized":"2024-01-16T13:52:06.382341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:04.918497+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:04.918198+0000","last_became_peered":"2024-01-16T13:52:04.918198+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:01.775696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904593+0000","last_change":"2024-01-16T13:52:03.904013+0000","last_active":"2024-01-16T13:52:05.904593+0000","last_peered":"2024-01-16T13:52:05.904593+0000","last_clean":"2024-01-16T13:52:05.904593+0000","last_became_active":"2024-01-16T13:52:03.903827+0000","last_became_peered":"2024-01-16T13:52:03.903827+0000","last_unstale":"2024-01-16T13:52:05.904593+0000","last_undegraded":"2024-01-16T13:52:05.904593+0000","last_fullsized":"2024-01-16T13:52:05.904593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383603+0000","last_change":"2024-01-16T13:52:04.912859+0000","last_active":"2024-01-16T13:52:06.383603+0000","last_peered":"2024-01-16T13:52:06.383603+0000","last_clean":"2024-01-16T13:52:06.383603+0000","last_became_active":"2024-01-16T13:52:04.912597+0000","last_became_peered":"2024-01-16T13:52:04.912597+0000","last_unstale":"2024-01-16T13:52:06.383603+0000","last_undegraded":"2024-01-16T13:52:06.383603+0000","last_fullsized":"2024-01-16T13:52:06.383603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:03.908118+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:03.907800+0000","last_became_peered":"2024-01-16T13:52:03.907800+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383308+0000","last_change":"2024-01-16T13:52:04.918302+0000","last_active":"2024-01-16T13:52:06.383308+0000","last_peered":"2024-01-16T13:52:06.383308+0000","last_clean":"2024-01-16T13:52:06.383308+0000","last_became_active":"2024-01-16T13:52:04.917994+0000","last_became_peered":"2024-01-16T13:52:04.917994+0000","last_unstale":"2024-01-16T13:52:06.383308+0000","last_undegraded":"2024-01-16T13:52:06.383308+0000","last_fullsized":"2024-01-16T13:52:06.383308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383067+0000","last_change":"2024-01-16T13:52:03.903338+0000","last_active":"2024-01-16T13:52:06.383067+0000","last_peered":"2024-01-16T13:52:06.383067+0000","last_clean":"2024-01-16T13:52:06.383067+0000","last_became_active":"2024-01-16T13:52:03.903053+0000","last_became_peered":"2024-01-16T13:52:03.903053+0000","last_unstale":"2024-01-16T13:52:06.383067+0000","last_undegraded":"2024-01-16T13:52:06.383067+0000","last_fullsized":"2024-01-16T13:52:06.383067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:27.027544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903790+0000","last_change":"2024-01-16T13:52:04.924256+0000","last_active":"2024-01-16T13:52:05.903790+0000","last_peered":"2024-01-16T13:52:05.903790+0000","last_clean":"2024-01-16T13:52:05.903790+0000","last_became_active":"2024-01-16T13:52:04.923945+0000","last_became_peered":"2024-01-16T13:52:04.923945+0000","last_unstale":"2024-01-16T13:52:05.903790+0000","last_undegraded":"2024-01-16T13:52:05.903790+0000","last_fullsized":"2024-01-16T13:52:05.903790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382587+0000","last_change":"2024-01-16T13:52:03.906252+0000","last_active":"2024-01-16T13:52:06.382587+0000","last_peered":"2024-01-16T13:52:06.382587+0000","last_clean":"2024-01-16T13:52:06.382587+0000","last_became_active":"2024-01-16T13:52:03.905949+0000","last_became_peered":"2024-01-16T13:52:03.905949+0000","last_unstale":"2024-01-16T13:52:06.382587+0000","last_undegraded":"2024-01-16T13:52:06.382587+0000","last_fullsized":"2024-01-16T13:52:06.382587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:00.771273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:04.933410+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:04.932914+0000","last_became_peered":"2024-01-16T13:52:04.932914+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904753+0000","last_change":"2024-01-16T13:52:03.897651+0000","last_active":"2024-01-16T13:52:05.904753+0000","last_peered":"2024-01-16T13:52:05.904753+0000","last_clean":"2024-01-16T13:52:05.904753+0000","last_became_active":"2024-01-16T13:52:03.897380+0000","last_became_peered":"2024-01-16T13:52:03.897380+0000","last_unstale":"2024-01-16T13:52:05.904753+0000","last_undegraded":"2024-01-16T13:52:05.904753+0000","last_fullsized":"2024-01-16T13:52:05.904753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904179+0000","last_change":"2024-01-16T13:52:04.934873+0000","last_active":"2024-01-16T13:52:05.904179+0000","last_peered":"2024-01-16T13:52:05.904179+0000","last_clean":"2024-01-16T13:52:05.904179+0000","last_became_active":"2024-01-16T13:52:04.934573+0000","last_became_peered":"2024-01-16T13:52:04.934573+0000","last_unstale":"2024-01-16T13:52:05.904179+0000","last_undegraded":"2024-01-16T13:52:05.904179+0000","last_fullsized":"2024-01-16T13:52:05.904179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:03.911272+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:03.910949+0000","last_became_peered":"2024-01-16T13:52:03.910949+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383514+0000","last_change":"2024-01-16T13:52:04.933704+0000","last_active":"2024-01-16T13:52:06.383514+0000","last_peered":"2024-01-16T13:52:06.383514+0000","last_clean":"2024-01-16T13:52:06.383514+0000","last_became_active":"2024-01-16T13:52:04.933390+0000","last_became_peered":"2024-01-16T13:52:04.933390+0000","last_unstale":"2024-01-16T13:52:06.383514+0000","last_undegraded":"2024-01-16T13:52:06.383514+0000","last_fullsized":"2024-01-16T13:52:06.383514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903638+0000","last_change":"2024-01-16T13:52:04.922828+0000","last_active":"2024-01-16T13:52:05.903638+0000","last_peered":"2024-01-16T13:52:05.903638+0000","last_clean":"2024-01-16T13:52:05.903638+0000","last_became_active":"2024-01-16T13:52:04.922428+0000","last_became_peered":"2024-01-16T13:52:04.922428+0000","last_unstale":"2024-01-16T13:52:05.903638+0000","last_undegraded":"2024-01-16T13:52:05.903638+0000","last_fullsized":"2024-01-16T13:52:05.903638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382275+0000","last_change":"2024-01-16T13:52:04.915322+0000","last_active":"2024-01-16T13:52:06.382275+0000","last_peered":"2024-01-16T13:52:06.382275+0000","last_clean":"2024-01-16T13:52:06.382275+0000","last_became_active":"2024-01-16T13:52:04.915031+0000","last_became_peered":"2024-01-16T13:52:04.915031+0000","last_unstale":"2024-01-16T13:52:06.382275+0000","last_undegraded":"2024-01-16T13:52:06.382275+0000","last_fullsized":"2024-01-16T13:52:06.382275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905142+0000","last_change":"2024-01-16T13:52:04.925541+0000","last_active":"2024-01-16T13:52:05.905142+0000","last_peered":"2024-01-16T13:52:05.905142+0000","last_clean":"2024-01-16T13:52:05.905142+0000","last_became_active":"2024-01-16T13:52:04.923411+0000","last_became_peered":"2024-01-16T13:52:04.923411+0000","last_unstale":"2024-01-16T13:52:05.905142+0000","last_undegraded":"2024-01-16T13:52:05.905142+0000","last_fullsized":"2024-01-16T13:52:05.905142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382903+0000","last_change":"2024-01-16T13:52:04.919996+0000","last_active":"2024-01-16T13:52:06.382903+0000","last_peered":"2024-01-16T13:52:06.382903+0000","last_clean":"2024-01-16T13:52:06.382903+0000","last_became_active":"2024-01-16T13:52:04.919257+0000","last_became_peered":"2024-01-16T13:52:04.919257+0000","last_unstale":"2024-01-16T13:52:06.382903+0000","last_undegraded":"2024-01-16T13:52:06.382903+0000","last_fullsized":"2024-01-16T13:52:06.382903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382991+0000","last_change":"2024-01-16T13:52:04.921215+0000","last_active":"2024-01-16T13:52:06.382991+0000","last_peered":"2024-01-16T13:52:06.382991+0000","last_clean":"2024-01-16T13:52:06.382991+0000","last_became_active":"2024-01-16T13:52:04.920992+0000","last_became_peered":"2024-01-16T13:52:04.920992+0000","last_unstale":"2024-01-16T13:52:06.382991+0000","last_undegraded":"2024-01-16T13:52:06.382991+0000","last_fullsized":"2024-01-16T13:52:06.382991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:04.911541+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:04.911263+0000","last_became_peered":"2024-01-16T13:52:04.911263+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904058+0000","last_change":"2024-01-16T13:52:04.932056+0000","last_active":"2024-01-16T13:52:05.904058+0000","last_peered":"2024-01-16T13:52:05.904058+0000","last_clean":"2024-01-16T13:52:05.904058+0000","last_became_active":"2024-01-16T13:52:04.931794+0000","last_became_peered":"2024-01-16T13:52:04.931794+0000","last_unstale":"2024-01-16T13:52:05.904058+0000","last_undegraded":"2024-01-16T13:52:05.904058+0000","last_fullsized":"2024-01-16T13:52:05.904058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383217+0000","last_change":"2024-01-16T13:52:04.918735+0000","last_active":"2024-01-16T13:52:06.383217+0000","last_peered":"2024-01-16T13:52:06.383217+0000","last_clean":"2024-01-16T13:52:06.383217+0000","last_became_active":"2024-01-16T13:52:04.918440+0000","last_became_peered":"2024-01-16T13:52:04.918440+0000","last_unstale":"2024-01-16T13:52:06.383217+0000","last_undegraded":"2024-01-16T13:52:06.383217+0000","last_fullsized":"2024-01-16T13:52:06.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905101+0000","last_change":"2024-01-16T13:52:04.923904+0000","last_active":"2024-01-16T13:52:05.905101+0000","last_peered":"2024-01-16T13:52:05.905101+0000","last_clean":"2024-01-16T13:52:05.905101+0000","last_became_active":"2024-01-16T13:52:04.923426+0000","last_became_peered":"2024-01-16T13:52:04.923426+0000","last_unstale":"2024-01-16T13:52:05.905101+0000","last_undegraded":"2024-01-16T13:52:05.905101+0000","last_fullsized":"2024-01-16T13:52:05.905101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383063+0000","last_change":"2024-01-16T13:52:04.919553+0000","last_active":"2024-01-16T13:52:06.383063+0000","last_peered":"2024-01-16T13:52:06.383063+0000","last_clean":"2024-01-16T13:52:06.383063+0000","last_became_active":"2024-01-16T13:52:04.919053+0000","last_became_peered":"2024-01-16T13:52:04.919053+0000","last_unstale":"2024-01-16T13:52:06.383063+0000","last_undegraded":"2024-01-16T13:52:06.383063+0000","last_fullsized":"2024-01-16T13:52:06.383063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905086+0000","last_change":"2024-01-16T13:52:04.926131+0000","last_active":"2024-01-16T13:52:05.905086+0000","last_peered":"2024-01-16T13:52:05.905086+0000","last_clean":"2024-01-16T13:52:05.905086+0000","last_became_active":"2024-01-16T13:52:04.925848+0000","last_became_peered":"2024-01-16T13:52:04.925848+0000","last_unstale":"2024-01-16T13:52:05.905086+0000","last_undegraded":"2024-01-16T13:52:05.905086+0000","last_fullsized":"2024-01-16T13:52:05.905086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382745+0000","last_change":"2024-01-16T13:52:04.931968+0000","last_active":"2024-01-16T13:52:06.382745+0000","last_peered":"2024-01-16T13:52:06.382745+0000","last_clean":"2024-01-16T13:52:06.382745+0000","last_became_active":"2024-01-16T13:52:04.931678+0000","last_became_peered":"2024-01-16T13:52:04.931678+0000","last_unstale":"2024-01-16T13:52:06.382745+0000","last_undegraded":"2024-01-16T13:52:06.382745+0000","last_fullsized":"2024-01-16T13:52:06.382745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904558+0000","last_change":"2024-01-16T13:52:04.929411+0000","last_active":"2024-01-16T13:52:05.904558+0000","last_peered":"2024-01-16T13:52:05.904558+0000","last_clean":"2024-01-16T13:52:05.904558+0000","last_became_active":"2024-01-16T13:52:04.929143+0000","last_became_peered":"2024-01-16T13:52:04.929143+0000","last_unstale":"2024-01-16T13:52:05.904558+0000","last_undegraded":"2024-01-16T13:52:05.904558+0000","last_fullsized":"2024-01-16T13:52:05.904558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:52:09.386 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T13:52:09.387 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T13:52:09.387 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T13:52:09.387 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T13:52:09.387 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T13:52:09.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.561+0000 7f4beb475700 1 Processor -- start 2024-01-16T13:52:09.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.562+0000 7f4beb475700 1 -- start start 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4beb475700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be40ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4beb475700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4be406ee50 0x7f4be4074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4beb475700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be4074e90 con 0x7f4be415c140 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4beb475700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be40ce5d0 con 0x7f4be406ee50 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4beb475700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40748b0 con 0x7f4be4073f90 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4be406ee50 0x7f4be4074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be9211700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be40ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4be406ee50 0x7f4be4074370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57574/0 (socket says 172.21.15.113:57574) 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 -- 172.21.15.113:0/3493983004 learned_addr learned my addr 172.21.15.113:0/3493983004 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be9a12700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f4be415c140 0x7f4be415f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:59098/0 (socket says 172.21.15.113:59098) 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 -- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 msgr2=0x7f4be40ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 --2- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be40ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 -- 172.21.15.113:0/3493983004 >> v1:172.21.15.124:6789/0 conn(0x7f4be415c140 legacy=0x7f4be415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 -- 172.21.15.113:0/3493983004 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be415c700 con 0x7f4be406ee50 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be9211700 1 --2- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be40ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4be8a10700 1 --2- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be4074370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098f0 tx=0x7f4be0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3382eda463749730 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4bdbfff700 1 -- 172.21.15.113:0/3493983004 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0011830 con 0x7f4be406ee50 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4bdbfff700 1 -- 172.21.15.113:0/3493983004 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be0011990 con 0x7f4be406ee50 2024-01-16T13:52:09.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.563+0000 7f4bdbfff700 1 -- 172.21.15.113:0/3493983004 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0011d20 con 0x7f4be406ee50 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 -- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 msgr2=0x7f4be4074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 --2- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be4074370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098f0 tx=0x7f4be0009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 -- 172.21.15.113:0/3493983004 shutdown_connections 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 --2- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be40ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 --2- 172.21.15.113:0/3493983004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be4074370 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 -- 172.21.15.113:0/3493983004 >> 172.21.15.113:0/3493983004 conn(0x7f4be40cd570 msgr2=0x7f4be40cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 -- 172.21.15.113:0/3493983004 shutdown_connections 2024-01-16T13:52:09.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.564+0000 7f4beb475700 1 -- 172.21.15.113:0/3493983004 wait complete. 2024-01-16T13:52:09.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.565+0000 7f4beb475700 1 Processor -- start 2024-01-16T13:52:09.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.565+0000 7f4beb475700 1 -- start start 2024-01-16T13:52:09.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4beb475700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be41e67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4beb475700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be413bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4beb475700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 0x7f4be413c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4beb475700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41e7260 con 0x7f4be415c140 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4beb475700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41e7540 con 0x7f4be406ee50 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4beb475700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41e7820 con 0x7f4be4073f90 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 0x7f4be413c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 0x7f4be413c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40354/0 (socket says 172.21.15.113:40354) 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 -- 172.21.15.113:0/4033930538 learned_addr learned my addr 172.21.15.113:0/4033930538 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be8a10700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be413bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9211700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be41e67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 -- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 msgr2=0x7f4be413bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be413bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 -- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 msgr2=0x7f4be41e67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be41e67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be9a12700 1 -- 172.21.15.113:0/4033930538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd40094a0 con 0x7f4be415c140 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.566+0000 7f4be8a10700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be413bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:09.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.567+0000 7f4be9a12700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 0x7f4be413c120 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00cbc0 tx=0x7f4bdc00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.567+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc00ab50 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.567+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc00acb0 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.567+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be0009550 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.567+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc007270 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.567+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be41e7de0 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.568+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be41e7fd0 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.568+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4bdc0073d0 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.569+0000 7f4bd9ffb700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4bc806b350 0x7f4bc806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.569+0000 7f4be9211700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4bc806b350 0x7f4bc806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.569+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bdc089820 con 0x7f4be415c140 2024-01-16T13:52:09.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.569+0000 7f4be9211700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4bc806b350 0x7f4bc806d810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009d00 tx=0x7f4bd4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.573+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bdc054740 con 0x7f4be415c140 2024-01-16T13:52:09.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.822+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4be4067f60 con 0x7f4be415c140 2024-01-16T13:52:09.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.825+0000 7f4bd9ffb700 1 -- 172.21.15.113:0/4033930538 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171564 (secure 0 0 0) 0x7f4bdc017070 con 0x7f4be415c140 2024-01-16T13:52:09.809 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:09.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4bc806b350 msgr2=0x7f4bc806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4bc806b350 0x7f4bc806d810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009d00 tx=0x7f4bd4008040 comp rx=0 tx=0).stop 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 msgr2=0x7f4be413c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 0x7f4be413c120 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00cbc0 tx=0x7f4bdc00cf80 comp rx=0 tx=0).stop 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 shutdown_connections 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4bc806b350 0x7f4bc806d810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4be415c140 0x7f4be413c120 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4be406ee50 0x7f4be41e67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 --2- 172.21.15.113:0/4033930538 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4be4073f90 0x7f4be413bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.827+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 >> 172.21.15.113:0/4033930538 conn(0x7f4be40cd570 msgr2=0x7f4be40b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.828+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 shutdown_connections 2024-01-16T13:52:09.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.828+0000 7f4beb475700 1 -- 172.21.15.113:0/4033930538 wait complete. 2024-01-16T13:52:09.821 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":5,"active_gid":4113,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":1582984142},{"type":"v1","addr":"172.21.15.113:6833","nonce":1582984142}]},"active_addr":"172.21.15.113:6833/1582984142","active_change":"2024-01-16T13:51:52.394408+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":1448441835}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":3308472332}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":2529293185}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":422055266}]}]} 2024-01-16T13:52:09.825 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T13:52:09.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T13:52:09.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.965+0000 7f0e71d39700 1 Processor -- start 2024-01-16T13:52:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.965+0000 7f0e71d39700 1 -- start start 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.966+0000 7f0e71d39700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.966+0000 7f0e71d39700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 0x7f0e6c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.966+0000 7f0e71d39700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0ae430 con 0x7f0e6c15e080 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.966+0000 7f0e71d39700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c0af0b0 con 0x7f0e6c0af4d0 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.966+0000 7f0e71d39700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0b05a0 con 0x7f0e6c06c710 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6affd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6bfff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 0x7f0e6c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6bfff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 0x7f0e6c0afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57590/0 (socket says 172.21.15.113:57590) 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6bfff700 1 -- 172.21.15.113:0/1281563800 learned_addr learned my addr 172.21.15.113:0/1281563800 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6affd700 1 -- 172.21.15.113:0/1281563800 >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 msgr2=0x7f0e6c0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6affd700 1 --2- 172.21.15.113:0/1281563800 >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 0x7f0e6c0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6affd700 1 -- 172.21.15.113:0/1281563800 >> v1:172.21.15.124:6789/0 conn(0x7f0e6c15e080 legacy=0x7f0e6c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6affd700 1 -- 172.21.15.113:0/1281563800 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0ae710 con 0x7f0e6c06c710 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6bfff700 1 --2- 172.21.15.113:0/1281563800 >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 0x7f0e6c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6affd700 1 --2- 172.21.15.113:0/1281563800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c0b5050 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009840 tx=0x7f0e5c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7068c5da1b897a90 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6a7fc700 1 -- 172.21.15.113:0/1281563800 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c011830 con 0x7f0e6c06c710 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6a7fc700 1 -- 172.21.15.113:0/1281563800 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c011990 con 0x7f0e6c06c710 2024-01-16T13:52:09.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.967+0000 7f0e6a7fc700 1 -- 172.21.15.113:0/1281563800 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c011d20 con 0x7f0e6c06c710 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 -- 172.21.15.113:0/1281563800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 msgr2=0x7f0e6c0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 --2- 172.21.15.113:0/1281563800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c0b5050 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009840 tx=0x7f0e5c009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 -- 172.21.15.113:0/1281563800 shutdown_connections 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 --2- 172.21.15.113:0/1281563800 >> v2:172.21.15.124:3300/0 conn(0x7f0e6c0af4d0 0x7f0e6c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 --2- 172.21.15.113:0/1281563800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c0b5050 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 -- 172.21.15.113:0/1281563800 >> 172.21.15.113:0/1281563800 conn(0x7f0e6c071b90 msgr2=0x7f0e6c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 -- 172.21.15.113:0/1281563800 shutdown_connections 2024-01-16T13:52:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.968+0000 7f0e71d39700 1 -- 172.21.15.113:0/1281563800 wait complete. 2024-01-16T13:52:09.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.969+0000 7f0e71d39700 1 Processor -- start 2024-01-16T13:52:09.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.969+0000 7f0e71d39700 1 -- start start 2024-01-16T13:52:09.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e71d39700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e71d39700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e6c0af4d0 0x7f0e6c146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e71d39700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 0x7f0e6c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e71d39700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c156e70 con 0x7f0e6c15e080 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e71d39700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c155890 con 0x7f0e6c0af4d0 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e71d39700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c155b70 con 0x7f0e6c06c710 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 0x7f0e6c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 0x7f0e6c15e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40368/0 (socket says 172.21.15.113:40368) 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 -- 172.21.15.113:0/3499551319 learned_addr learned my addr 172.21.15.113:0/3499551319 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6affd700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e6c0af4d0 0x7f0e6c146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6b7fe700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 -- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 msgr2=0x7f0e6c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 -- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e6c0af4d0 msgr2=0x7f0e6c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e6c0af4d0 0x7f0e6c146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 -- 172.21.15.113:0/3499551319 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e64009b30 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e6bfff700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 0x7f0e6c15e7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400efc0 tx=0x7f0e6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.970+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64018070 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.971+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6400f040 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.971+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c0094a0 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.971+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64003c20 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.971+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c156130 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.971+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c0753f0 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.972+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0e6400c880 con 0x7f0e6c15e080 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.972+0000 7f0e63fff700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e4c06b350 0x7f0e4c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.972+0000 7f0e6b7fe700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e4c06b350 0x7f0e4c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.972+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e64089d70 con 0x7f0e6c15e080 2024-01-16T13:52:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.973+0000 7f0e6b7fe700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e4c06b350 0x7f0e4c06d810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007fd0 tx=0x7f0e54009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:09.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:09.977+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e64054bd0 con 0x7f0e6c15e080 2024-01-16T13:52:10.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.168+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e6c067f60 con 0x7f0e6c15e080 2024-01-16T13:52:10.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.169+0000 7f0e63fff700 1 -- 172.21.15.113:0/3499551319 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f0e640581f0 con 0x7f0e6c15e080 2024-01-16T13:52:10.154 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:10.154 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e4c06b350 msgr2=0x7f0e4c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e4c06b350 0x7f0e4c06d810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007fd0 tx=0x7f0e54009040 comp rx=0 tx=0).stop 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 msgr2=0x7f0e6c15e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 0x7f0e6c15e7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400efc0 tx=0x7f0e6400c3a0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 shutdown_connections 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e4c06b350 0x7f0e4c06d810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e6c15e080 0x7f0e6c15e7f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e6c06c710 0x7f0e6c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 --2- 172.21.15.113:0/3499551319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e6c0af4d0 0x7f0e6c146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.171+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 >> 172.21.15.113:0/3499551319 conn(0x7f0e6c071b90 msgr2=0x7f0e6c0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.172+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 shutdown_connections 2024-01-16T13:52:10.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.172+0000 7f0e71d39700 1 -- 172.21.15.113:0/3499551319 wait complete. 2024-01-16T13:52:10.166 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T13:52:10.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:10.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.305+0000 7fd60cdcf700 1 Processor -- start 2024-01-16T13:52:10.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.305+0000 7fd60cdcf700 1 -- start start 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60cdcf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd60815c160 0x7fd6080622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60cdcf700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd6080b04b0 0x7fd608165c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60cdcf700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60815c540 con 0x7fd6080b2870 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60cdcf700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6080b2cf0 con 0x7fd6080b04b0 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60cdcf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6080b2ee0 con 0x7fd60815c160 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd605d9b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fd6080b2870 0x7fd6080afda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:51992/0 (socket says 172.21.15.113:51992) 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd605d9b700 1 -- 172.21.15.113:0/3331398078 learned_addr learned my addr 172.21.15.113:0/3331398078 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd606d9d700 1 --2- 172.21.15.113:0/3331398078 >> v2:172.21.15.124:3300/0 conn(0x7fd6080b04b0 0x7fd608165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60659c700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd60815c160 0x7fd6080622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60559a700 1 -- 172.21.15.113:0/3331398078 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310103556 0 0) 0x7fd60815c540 con 0x7fd6080b2870 2024-01-16T13:52:10.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60559a700 1 -- 172.21.15.113:0/3331398078 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f00035b0 con 0x7fd6080b2870 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60559a700 1 -- 172.21.15.113:0/3331398078 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785478593 0 0) 0x7fd5f00035b0 con 0x7fd6080b2870 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd606d9d700 1 -- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd60815c160 msgr2=0x7fd6080622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd606d9d700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd60815c160 0x7fd6080622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd606d9d700 1 -- 172.21.15.113:0/3331398078 >> v1:172.21.15.113:6789/0 conn(0x7fd6080b2870 legacy=0x7fd6080afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd606d9d700 1 -- 172.21.15.113:0/3331398078 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080add40 con 0x7fd6080b04b0 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd60659c700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd60815c160 0x7fd6080622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.306+0000 7fd606d9d700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd6080b04b0 0x7fd608165c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd600009d90 tx=0x7fd60000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71d03b1b7471e570 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60559a700 1 -- 172.21.15.113:0/3331398078 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60000e070 con 0x7fd6080b04b0 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60559a700 1 -- 172.21.15.113:0/3331398078 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd600003a20 con 0x7fd6080b04b0 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60559a700 1 -- 172.21.15.113:0/3331398078 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc001670 con 0x7fd6080b04b0 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 -- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd6080b04b0 msgr2=0x7fd608165c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd6080b04b0 0x7fd608165c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd600009d90 tx=0x7fd60000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 -- 172.21.15.113:0/3331398078 shutdown_connections 2024-01-16T13:52:10.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd6080b04b0 0x7fd608165c80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/3331398078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd60815c160 0x7fd6080622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 -- 172.21.15.113:0/3331398078 >> 172.21.15.113:0/3331398078 conn(0x7fd6080b57b0 msgr2=0x7fd6080b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 -- 172.21.15.113:0/3331398078 shutdown_connections 2024-01-16T13:52:10.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.307+0000 7fd60cdcf700 1 -- 172.21.15.113:0/3331398078 wait complete. 2024-01-16T13:52:10.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 Processor -- start 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 -- start start 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd6080b04b0 0x7fd6080c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 0x7fd6081f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd60815c160 0x7fd6081f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60815f400 con 0x7fd6080b2870 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60815e570 con 0x7fd60815c160 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60cdcf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6081f3740 con 0x7fd6080b04b0 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd605d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 0x7fd6081f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd605d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 0x7fd6081f2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40384/0 (socket says 172.21.15.113:40384) 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd605d9b700 1 -- 172.21.15.113:0/42607309 learned_addr learned my addr 172.21.15.113:0/42607309 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.309+0000 7fd60659c700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd6080b04b0 0x7fd6080c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd606d9d700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd60815c160 0x7fd6081f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd605d9b700 1 -- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd6080b04b0 msgr2=0x7fd6080c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd605d9b700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd6080b04b0 0x7fd6080c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd605d9b700 1 -- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd60815c160 msgr2=0x7fd6081f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd605d9b700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd60815c160 0x7fd6081f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd605d9b700 1 -- 172.21.15.113:0/42607309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f80094a0 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd605d9b700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 0x7fd6081f2cc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c780 tx=0x7fd5fc00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc004c80 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc004de0 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd600009a70 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0072f0 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.310+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60815ed40 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.311+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60815f540 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.311+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd5fc007450 con 0x7fd6080b2870 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.312+0000 7fd5f6ffd700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd5e806b350 0x7fd5e806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.312+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5fc0898f0 con 0x7fd6080b2870 2024-01-16T13:52:10.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.312+0000 7fd60659c700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd5e806b350 0x7fd5e806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.312+0000 7fd60659c700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd5e806b350 0x7fd5e806d810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009cd0 tx=0x7fd5f8009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.316+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd60815f540 con 0x7fd6080b2870 2024-01-16T13:52:10.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:52:10.409 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:52:10.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.515+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd60815f540 con 0x7fd6080b2870 2024-01-16T13:52:10.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.516+0000 7fd5f6ffd700 1 -- 172.21.15.113:0/42607309 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fd60815f540 con 0x7fd6080b2870 2024-01-16T13:52:10.499 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:10.499 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd5e806b350 msgr2=0x7fd5e806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd5e806b350 0x7fd5e806d810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009cd0 tx=0x7fd5f8009210 comp rx=0 tx=0).stop 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 msgr2=0x7fd6081f2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 0x7fd6081f2cc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c780 tx=0x7fd5fc00ca90 comp rx=0 tx=0).stop 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 shutdown_connections 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd5e806b350 0x7fd5e806d810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd60815c160 0x7fd6081f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd6080b04b0 0x7fd6080c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 --2- 172.21.15.113:0/42607309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6080b2870 0x7fd6081f2cc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 >> 172.21.15.113:0/42607309 conn(0x7fd6080b57b0 msgr2=0x7fd6080ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 shutdown_connections 2024-01-16T13:52:10.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.519+0000 7fd60cdcf700 1 -- 172.21.15.113:0/42607309 wait complete. 2024-01-16T13:52:10.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T13:52:10.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T13:52:10.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T13:52:10.514 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-16T13:52:10.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-16T13:52:10.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-16T13:52:10.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-16T13:52:10.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-16T13:52:10.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.783+0000 7fe84634e700 1 Processor -- start 2024-01-16T13:52:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.784+0000 7f4e68b85700 1 Processor -- start 2024-01-16T13:52:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.783+0000 7fe84634e700 1 -- start start 2024-01-16T13:52:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.784+0000 7f4e68b85700 1 -- start start 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7ff3286e6700 1 Processor -- start 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe84634e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe8400cf440 0x7fe84006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe84634e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 0x7fe840165e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe84634e700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe840072110 con 0x7fe8401663d0 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe84634e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe840166860 con 0x7fe8400cf440 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe84634e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84006ee00 con 0x7fe840070fd0 2024-01-16T13:52:10.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f7f398aa700 1 Processor -- start 2024-01-16T13:52:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f4e68b85700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4e6415af10 0x7f4e6415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f4e68b85700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4e640ce140 0x7f4e6415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f4e68b85700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e641414c0 con 0x7f4e6415f460 2024-01-16T13:52:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f4e68b85700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e640ce9f0 con 0x7f4e6415af10 2024-01-16T13:52:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f4e68b85700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e640cecd0 con 0x7f4e640ce140 2024-01-16T13:52:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe83f7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 0x7fe840165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe83f7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 0x7fe840165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36666/0 (socket says 172.21.15.113:36666) 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7fe83f7fe700 1 -- 172.21.15.113:0/2645683072 learned_addr learned my addr 172.21.15.113:0/2645683072 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e62d9d700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f4e6415f460 0x7f4e6415b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:59110/0 (socket says 172.21.15.113:59110) 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e62d9d700 1 -- 172.21.15.113:0/1040000162 learned_addr learned my addr 172.21.15.113:0/1040000162 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e61d9b700 1 --2- 172.21.15.113:0/1040000162 >> v2:172.21.15.124:3300/0 conn(0x7f4e640ce140 0x7f4e6415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7fe83ffff700 1 --2- 172.21.15.113:0/2645683072 >> v2:172.21.15.113:3300/0 conn(0x7fe8400cf440 0x7fe84006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7fe83f7fe700 1 -- 172.21.15.113:0/2645683072 >> v2:172.21.15.113:3300/0 conn(0x7fe8400cf440 msgr2=0x7fe84006e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7f7f398aa700 1 -- start start 2024-01-16T13:52:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7fe83f7fe700 1 --2- 172.21.15.113:0/2645683072 >> v2:172.21.15.113:3300/0 conn(0x7fe8400cf440 0x7fe84006e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7fe83f7fe700 1 -- 172.21.15.113:0/2645683072 >> v1:172.21.15.113:6789/0 conn(0x7fe8401663d0 legacy=0x7fe8401628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7fe83f7fe700 1 -- 172.21.15.113:0/2645683072 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840071500 con 0x7fe840070fd0 2024-01-16T13:52:10.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.785+0000 7ff3286e6700 1 -- start start 2024-01-16T13:52:10.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7fe83f7fe700 1 --2- 172.21.15.113:0/2645683072 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 0x7fe840165e90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe8340098f0 tx=0x7fe834009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61b00a4852277d42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e61d9b700 1 -- 172.21.15.113:0/1040000162 >> v2:172.21.15.113:3300/0 conn(0x7f4e6415af10 msgr2=0x7f4e6415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e6159a700 1 -- 172.21.15.113:0/1040000162 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849484874 0 0) 0x7f4e641414c0 con 0x7f4e6415f460 2024-01-16T13:52:10.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e61d9b700 1 --2- 172.21.15.113:0/1040000162 >> v2:172.21.15.113:3300/0 conn(0x7f4e6415af10 0x7f4e6415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e61d9b700 1 -- 172.21.15.113:0/1040000162 >> v1:172.21.15.124:6789/0 conn(0x7f4e6415f460 legacy=0x7f4e6415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.786+0000 7f4e61d9b700 1 -- 172.21.15.113:0/1040000162 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6406b120 con 0x7f4e640ce140 2024-01-16T13:52:10.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff3286e6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff320162240 0x7ff3200afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f4e61d9b700 1 --2- 172.21.15.113:0/1040000162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e640ce140 0x7f4e6415ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e6406d090 tx=0x7f4e54009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7988e87130870a38 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7fe83effd700 1 -- 172.21.15.113:0/2645683072 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83400b8a0 con 0x7fe840070fd0 2024-01-16T13:52:10.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7fe83ffff700 1 --2- 172.21.15.113:0/2645683072 >> v2:172.21.15.113:3300/0 conn(0x7fe8400cf440 0x7fe84006e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:10.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff3286e6700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff3200ae4e0 0x7ff32006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff3286e6700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3200b46e0 con 0x7ff32006f360 2024-01-16T13:52:10.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff3286e6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff320166f50 con 0x7ff320162240 2024-01-16T13:52:10.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff3286e6700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3200b49c0 con 0x7ff3200ae4e0 2024-01-16T13:52:10.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7fe83effd700 1 -- 172.21.15.113:0/2645683072 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83400ba00 con 0x7fe840070fd0 2024-01-16T13:52:10.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7fe83effd700 1 -- 172.21.15.113:0/2645683072 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83400bd90 con 0x7fe840070fd0 2024-01-16T13:52:10.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f4e6159a700 1 -- 172.21.15.113:0/1040000162 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54011830 con 0x7f4e640ce140 2024-01-16T13:52:10.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f4e6159a700 1 -- 172.21.15.113:0/1040000162 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e54011990 con 0x7f4e640ce140 2024-01-16T13:52:10.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f4e6159a700 1 -- 172.21.15.113:0/1040000162 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54011d20 con 0x7f4e640ce140 2024-01-16T13:52:10.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff320162240 0x7ff3200afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff320162240 0x7ff3200afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40400/0 (socket says 172.21.15.113:40400) 2024-01-16T13:52:10.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 -- 172.21.15.113:0/807580988 learned_addr learned my addr 172.21.15.113:0/807580988 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f398aa700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7f3415d6a0 0x7f7f340b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f398aa700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7f34071be0 0x7f7f34140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f398aa700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f340720e0 con 0x7f7f340b4090 2024-01-16T13:52:10.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f398aa700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f34072d60 con 0x7f7f34071be0 2024-01-16T13:52:10.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f398aa700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3406de80 con 0x7f7f3415d6a0 2024-01-16T13:52:10.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f32ffd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7f3415d6a0 0x7f7f340b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f32ffd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7f3415d6a0 0x7f7f340b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57634/0 (socket says 172.21.15.113:57634) 2024-01-16T13:52:10.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7f7f32ffd700 1 -- 172.21.15.113:0/3970882249 learned_addr learned my addr 172.21.15.113:0/3970882249 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 -- 172.21.15.113:0/807580988 >> v2:172.21.15.124:3300/0 conn(0x7ff3200ae4e0 msgr2=0x7ff32006c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:10.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 --2- 172.21.15.113:0/807580988 >> v2:172.21.15.124:3300/0 conn(0x7ff3200ae4e0 0x7ff32006c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 -- 172.21.15.113:0/807580988 >> v1:172.21.15.113:6789/0 conn(0x7ff32006f360 legacy=0x7ff320073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.787+0000 7ff326482700 1 -- 172.21.15.113:0/807580988 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3200b4ca0 con 0x7ff320162240 2024-01-16T13:52:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7ff326482700 1 --2- 172.21.15.113:0/807580988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff320162240 0x7ff3200afed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3200b57b0 tx=0x7ff31c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7795a6ec271cb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7fe84634e700 1 -- 172.21.15.113:0/2645683072 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 msgr2=0x7fe840165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7fe84634e700 1 --2- 172.21.15.113:0/2645683072 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 0x7fe840165e90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe8340098f0 tx=0x7fe834009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:10.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7fd0a7ec3700 1 Processor -- start 2024-01-16T13:52:10.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7f4e68b85700 1 -- 172.21.15.113:0/1040000162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e640ce140 msgr2=0x7f4e6415ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7f7f32ffd700 1 -- 172.21.15.113:0/3970882249 >> v2:172.21.15.113:3300/0 conn(0x7f7f34071be0 msgr2=0x7f7f34140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7f7f32ffd700 1 --2- 172.21.15.113:0/3970882249 >> v2:172.21.15.113:3300/0 conn(0x7f7f34071be0 0x7f7f34140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7f4e68b85700 1 --2- 172.21.15.113:0/1040000162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e640ce140 0x7f4e6415ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e6406d090 tx=0x7f4e54009b20 comp rx=0 tx=0).stop 2024-01-16T13:52:10.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7f7f32ffd700 1 -- 172.21.15.113:0/3970882249 >> v1:172.21.15.124:6789/0 conn(0x7f7f340b4090 legacy=0x7f7f340714d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:10.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7f7f32ffd700 1 -- 172.21.15.113:0/3970882249 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340723c0 con 0x7f7f3415d6a0 2024-01-16T13:52:10.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f17a6acc700 1 Processor -- start 2024-01-16T13:52:10.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7ff325480700 1 -- 172.21.15.113:0/807580988 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00b870 con 0x7ff320162240 2024-01-16T13:52:10.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7ff325480700 1 -- 172.21.15.113:0/807580988 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c00b9d0 con 0x7ff320162240 2024-01-16T13:52:10.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7ff325480700 1 -- 172.21.15.113:0/807580988 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00bd60 con 0x7ff320162240 2024-01-16T13:52:10.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f7f32ffd700 1 --2- 172.21.15.113:0/3970882249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 0x7f7f340b3b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7f340cfa00 tx=0x7f7f2c009730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0abbf67f49a4e79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f2bf1af7700 1 Processor -- start 2024-01-16T13:52:10.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.788+0000 7fd0a7ec3700 1 -- start start 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f4e68b85700 1 -- 172.21.15.113:0/1040000162 shutdown_connections 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f4e68b85700 1 --2- 172.21.15.113:0/1040000162 >> v2:172.21.15.113:3300/0 conn(0x7f4e6415af10 0x7f4e6415e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f4e68b85700 1 --2- 172.21.15.113:0/1040000162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e640ce140 0x7f4e6415ef20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f4e68b85700 1 -- 172.21.15.113:0/1040000162 >> 172.21.15.113:0/1040000162 conn(0x7f4e640cd570 msgr2=0x7f4e640cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f4e68b85700 1 -- 172.21.15.113:0/1040000162 shutdown_connections 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7fe84634e700 1 -- 172.21.15.113:0/2645683072 shutdown_connections 2024-01-16T13:52:10.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7fe84634e700 1 --2- 172.21.15.113:0/2645683072 >> v2:172.21.15.113:3300/0 conn(0x7fe8400cf440 0x7fe84006e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7fe84634e700 1 --2- 172.21.15.113:0/2645683072 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe840070fd0 0x7fe840165e90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7fe84634e700 1 -- 172.21.15.113:0/2645683072 >> 172.21.15.113:0/2645683072 conn(0x7fe8400b1740 msgr2=0x7fe8400b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7fe84634e700 1 -- 172.21.15.113:0/2645683072 shutdown_connections 2024-01-16T13:52:10.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7ff3286e6700 1 -- 172.21.15.113:0/807580988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff320162240 msgr2=0x7ff3200afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7ff3286e6700 1 --2- 172.21.15.113:0/807580988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff320162240 0x7ff3200afed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3200b57b0 tx=0x7ff31c009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a7ec3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 0x7fd0a00d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a7ec3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd0a00d3a00 0x7fd0a00d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a7ec3700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a00d20b0 con 0x7fd0a00d0db0 2024-01-16T13:52:10.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a7ec3700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a00d3150 con 0x7fd0a00d3a00 2024-01-16T13:52:10.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a7ec3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00d1240 con 0x7fd0a015e1e0 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f7f31ffb700 1 -- 172.21.15.113:0/3970882249 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c003c20 con 0x7f7f3415d6a0 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f7f31ffb700 1 -- 172.21.15.113:0/3970882249 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c003d80 con 0x7f7f3415d6a0 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f7f31ffb700 1 -- 172.21.15.113:0/3970882249 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c00c930 con 0x7f7f3415d6a0 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 -- 172.21.15.113:0/3970882249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 msgr2=0x7f7f340b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 --2- 172.21.15.113:0/3970882249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 0x7f7f340b3b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7f340cfa00 tx=0x7f7f2c009730 comp rx=0 tx=0).stop 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f17a6acc700 1 -- start start 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f17a6acc700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f17a00b4260 0x7f17a00714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f17a6acc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a0071a10 0x7f17a0140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f17a6acc700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a0071e90 con 0x7f17a015d6a0 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f17a6acc700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a0072d60 con 0x7f17a00b4260 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f17a6acc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a006de80 con 0x7f17a0071a10 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f179ffff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f17a00b4260 0x7f17a00714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f179ffff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f17a00b4260 0x7f17a00714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57648/0 (socket says 172.21.15.113:57648) 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f179ffff700 1 -- 172.21.15.113:0/3701765453 learned_addr learned my addr 172.21.15.113:0/3701765453 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f17a5069700 1 --2- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a0071a10 0x7f17a0140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f179ffff700 1 -- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a0071a10 msgr2=0x7f17a0140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f179ffff700 1 --2- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a0071a10 0x7f17a0140fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f179ffff700 1 -- 172.21.15.113:0/3701765453 >> v1:172.21.15.113:6789/0 conn(0x7f17a015d6a0 legacy=0x7f17a00b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f179ffff700 1 -- 172.21.15.113:0/3701765453 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00720e0 con 0x7f17a00b4260 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f179ffff700 1 --2- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a00b4260 0x7f17a00714d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f179800c310 tx=0x7f179800c620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d5d776e7123c9de server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f179f7fe700 1 -- 172.21.15.113:0/3701765453 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179800f070 con 0x7f17a00b4260 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f179f7fe700 1 -- 172.21.15.113:0/3701765453 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1798003a20 con 0x7f17a00b4260 2024-01-16T13:52:10.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f179f7fe700 1 -- 172.21.15.113:0/3701765453 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798003db0 con 0x7f17a00b4260 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f2bf1af7700 1 -- start start 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f2bf1af7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2bec0b7430 0x7f2bec163580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f2bf1af7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0d3460 con 0x7f2bec0d3080 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f2bf1af7700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0b3390 con 0x7f2bec0b9bf0 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f2bf1af7700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bec0b3170 con 0x7f2bec0b7430 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f2beaffd700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f2bec0b9bf0 0x7f2bec0b6d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:59118/0 (socket says 172.21.15.113:59118) 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f2beaffd700 1 -- 172.21.15.113:0/340061762 learned_addr learned my addr 172.21.15.113:0/340061762 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f2bebfff700 1 --2- 172.21.15.113:0/340061762 >> v2:172.21.15.113:3300/0 conn(0x7f2bec0b7430 0x7f2bec163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f2bebfff700 1 -- 172.21.15.113:0/340061762 >> v1:172.21.15.124:6789/0 conn(0x7f2bec0b9bf0 legacy=0x7f2bec0b6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f2bebfff700 1 -- 172.21.15.113:0/340061762 >> v1:172.21.15.113:6789/0 conn(0x7f2bec0d3080 legacy=0x7f2bec0b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f2bebfff700 1 -- 172.21.15.113:0/340061762 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec162aa0 con 0x7f2bec0b7430 2024-01-16T13:52:10.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f2bebfff700 1 --2- 172.21.15.113:0/340061762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0b7430 0x7f2bec163580 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009dc0 tx=0x7f2be400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8cfc6971eca889 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a5c5f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 0x7fd0a00d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a545e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd0a00d3a00 0x7fd0a00d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a5c5f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 0x7fd0a00d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36674/0 (socket says 172.21.15.113:36674) 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/171085011 learned_addr learned my addr 172.21.15.113:0/171085011 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7fd0a545e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd0a00d3a00 0x7fd0a00d0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57636/0 (socket says 172.21.15.113:57636) 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/171085011 >> v2:172.21.15.124:3300/0 conn(0x7fd0a00d3a00 msgr2=0x7fd0a00d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7fd0a5c5f700 1 --2- 172.21.15.113:0/171085011 >> v2:172.21.15.124:3300/0 conn(0x7fd0a00d3a00 0x7fd0a00d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/171085011 >> v1:172.21.15.113:6789/0 conn(0x7fd0a00d0db0 legacy=0x7fd0a006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/171085011 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00d2390 con 0x7fd0a015e1e0 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7fd0a5c5f700 1 --2- 172.21.15.113:0/171085011 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 0x7fd0a00d34c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098f0 tx=0x7fd094009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf7fd6fbe5fa7e2d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fd0a4c5d700 1 -- 172.21.15.113:0/171085011 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094004230 con 0x7fd0a015e1e0 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fd0a4c5d700 1 -- 172.21.15.113:0/171085011 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd094004390 con 0x7fd0a015e1e0 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fd0a4c5d700 1 -- 172.21.15.113:0/171085011 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0940036a0 con 0x7fd0a015e1e0 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/171085011 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 msgr2=0x7fd0a00d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/171085011 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 0x7fd0a00d34c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098f0 tx=0x7fd094009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7fe84634e700 1 -- 172.21.15.113:0/2645683072 wait complete. 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7fe84634e700 1 Processor -- start 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7fe84634e700 1 -- start start 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe84634e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe840070fd0 0x7fe840152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe84634e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe8400cf440 0x7fe84015fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe84634e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8401663d0 0x7fe840143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe84634e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe840164410 con 0x7fe8401663d0 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe84634e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe840164cb0 con 0x7fe840070fd0 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe84634e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe840152c70 con 0x7fe8400cf440 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe83f7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe8400cf440 0x7fe84015fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe83f7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe8400cf440 0x7fe84015fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36706/0 (socket says 172.21.15.113:36706) 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe83f7fe700 1 -- 172.21.15.113:0/2098609986 learned_addr learned my addr 172.21.15.113:0/2098609986 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe83ffff700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe840070fd0 0x7fe840152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7fe8448eb700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8401663d0 0x7fe840143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 -- 172.21.15.113:0/3970882249 shutdown_connections 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 --2- 172.21.15.113:0/3970882249 >> v2:172.21.15.113:3300/0 conn(0x7f7f34071be0 0x7f7f34140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 --2- 172.21.15.113:0/3970882249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 0x7f7f340b3b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 -- 172.21.15.113:0/3970882249 >> 172.21.15.113:0/3970882249 conn(0x7f7f340cfa30 msgr2=0x7f7f340cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 -- 172.21.15.113:0/3970882249 shutdown_connections 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.790+0000 7f7f398aa700 1 -- 172.21.15.113:0/3970882249 wait complete. 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f7f398aa700 1 Processor -- start 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f7f398aa700 1 -- start start 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f398aa700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f34071be0 0x7f7f3413ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f398aa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f340b4090 0x7f7f340ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f398aa700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 0x7f7f340ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f398aa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f341544a0 con 0x7f7f340b4090 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f398aa700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340cf280 con 0x7f7f3415d6a0 2024-01-16T13:52:10.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f398aa700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3413b060 con 0x7f7f34071be0 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.789+0000 7f4e68b85700 1 -- 172.21.15.113:0/1040000162 wait complete. 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f4e68b85700 1 Processor -- start 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.791+0000 7f4e68b85700 1 -- start start 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.792+0000 7f4e68b85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e640ce140 0x7f4e640b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e68b85700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e6415af10 0x7f4e6413eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e68b85700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4e6415f460 0x7f4e6413d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e68b85700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64158830 con 0x7f4e640ce140 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e68b85700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64167d50 con 0x7f4e6415af10 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e68b85700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e641674b0 con 0x7f4e6415f460 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4e6415f460 0x7f4e6413d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4e6415f460 0x7f4e6413d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36690/0 (socket says 172.21.15.113:36690) 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 -- 172.21.15.113:0/2103859906 learned_addr learned my addr 172.21.15.113:0/2103859906 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e6259c700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e640ce140 0x7f4e640b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e61d9b700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e6415af10 0x7f4e6413eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 -- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e6415af10 msgr2=0x7f4e6413eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4e6415af10 0x7f4e6413eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 -- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e640ce140 msgr2=0x7f4e640b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e640ce140 0x7f4e640b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f4e62d9d700 1 -- 172.21.15.113:0/2103859906 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e540094a0 con 0x7f4e6415f460 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.793+0000 7f7f32ffd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f34071be0 0x7f7f3413ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/171085011 shutdown_connections 2024-01-16T13:52:10.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/171085011 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a015e1e0 0x7fd0a00d34c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/171085011 >> v2:172.21.15.124:3300/0 conn(0x7fd0a00d3a00 0x7fd0a00d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/171085011 >> 172.21.15.113:0/171085011 conn(0x7fd0a00b5b50 msgr2=0x7fd0a00b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fe83ffff700 1 -- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe8400cf440 msgr2=0x7fe84015fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fe83ffff700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe8400cf440 0x7fe84015fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fe83ffff700 1 -- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8401663d0 msgr2=0x7fe840143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fe83ffff700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8401663d0 0x7fe840143900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fe83ffff700 1 -- 172.21.15.113:0/2098609986 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834009550 con 0x7fe840070fd0 2024-01-16T13:52:10.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f4e62d9d700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4e6415f460 0x7f4e6413d0b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00cbd0 tx=0x7f4e5c00cee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 -- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a00b4260 msgr2=0x7f17a00714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 --2- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a00b4260 0x7f17a00714d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f179800c310 tx=0x7f179800c620 comp rx=0 tx=0).stop 2024-01-16T13:52:10.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/171085011 shutdown_connections 2024-01-16T13:52:10.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00aa60 con 0x7f4e6415f460 2024-01-16T13:52:10.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c00abe0 con 0x7f4e6415f460 2024-01-16T13:52:10.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c0070d0 con 0x7f4e6415f460 2024-01-16T13:52:10.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f337fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 0x7f7f340ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7ff3286e6700 1 -- 172.21.15.113:0/807580988 shutdown_connections 2024-01-16T13:52:10.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7ff3286e6700 1 --2- 172.21.15.113:0/807580988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff320162240 0x7ff3200afed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7ff3286e6700 1 --2- 172.21.15.113:0/807580988 >> v2:172.21.15.124:3300/0 conn(0x7ff3200ae4e0 0x7ff32006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7ff3286e6700 1 -- 172.21.15.113:0/807580988 >> 172.21.15.113:0/807580988 conn(0x7ff320071b90 msgr2=0x7ff320071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 -- 172.21.15.113:0/3701765453 shutdown_connections 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 --2- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a0071a10 0x7f17a0140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 --2- 172.21.15.113:0/3701765453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a00b4260 0x7f17a00714d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f34071be0 0x7f7f3413ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36698/0 (socket says 172.21.15.113:36698) 2024-01-16T13:52:10.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 -- 172.21.15.113:0/282956461 learned_addr learned my addr 172.21.15.113:0/282956461 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 -- 172.21.15.113:0/3701765453 >> 172.21.15.113:0/3701765453 conn(0x7f17a00cfa30 msgr2=0x7f17a00cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 -- 172.21.15.113:0/3701765453 shutdown_connections 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f17a6acc700 1 -- 172.21.15.113:0/3701765453 wait complete. 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f17a6acc700 1 Processor -- start 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f17a6acc700 1 -- start start 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f17a6acc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a0071a10 0x7f17a01f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f17a6acc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a00b4260 0x7f17a01e5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f17a6acc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17a015d6a0 0x7f17a01e3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f17a6acc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a01e6870 con 0x7f17a015d6a0 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f17a6acc700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a01e5cf0 con 0x7f17a0071a10 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f17a6acc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a01e5fd0 con 0x7f17a00b4260 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a5069700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17a015d6a0 0x7f17a01e3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a0071a10 0x7f17a01f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a5069700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17a015d6a0 0x7f17a01e3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40482/0 (socket says 172.21.15.113:40482) 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a5069700 1 -- 172.21.15.113:0/3827002357 learned_addr learned my addr 172.21.15.113:0/3827002357 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f179ffff700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a00b4260 0x7f17a01e5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.795+0000 7f2bea7fc700 1 -- 172.21.15.113:0/340061762 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be400e070 con 0x7f2bec0b7430 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.795+0000 7f2bea7fc700 1 -- 172.21.15.113:0/340061762 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be400bad0 con 0x7f2bec0b7430 2024-01-16T13:52:10.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.795+0000 7f2bea7fc700 1 -- 172.21.15.113:0/340061762 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be400be60 con 0x7f2bec0b7430 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 -- 172.21.15.113:0/340061762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0b7430 msgr2=0x7f2bec163580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 --2- 172.21.15.113:0/340061762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0b7430 0x7f2bec163580 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009dc0 tx=0x7f2be400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 -- 172.21.15.113:0/340061762 shutdown_connections 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 --2- 172.21.15.113:0/340061762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0b7430 0x7f2bec163580 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 -- 172.21.15.113:0/340061762 >> 172.21.15.113:0/340061762 conn(0x7f2bec0d3ed0 msgr2=0x7f2bec0d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 -- 172.21.15.113:0/340061762 shutdown_connections 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f2bf1af7700 1 -- 172.21.15.113:0/340061762 wait complete. 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/171085011 wait complete. 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.795+0000 7fd0a7ec3700 1 Processor -- start 2024-01-16T13:52:10.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.795+0000 7fd0a7ec3700 1 -- start start 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7fd0a7ec3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a00d0db0 0x7fd0a0171c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7fd0a7ec3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0a00d3a00 0x7fd0a00cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7fd0a7ec3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd0a015e1e0 0x7fd0a006edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7fd0a7ec3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00cd3d0 con 0x7fd0a00d3a00 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7fd0a7ec3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a01721c0 con 0x7fd0a015e1e0 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7fd0a7ec3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00cc820 con 0x7fd0a00d0db0 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a00d0db0 0x7fd0a0171c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a00d0db0 0x7fd0a0171c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36722/0 (socket says 172.21.15.113:36722) 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/2322222668 learned_addr learned my addr 172.21.15.113:0/2322222668 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/2322222668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd0a015e1e0 msgr2=0x7fd0a006edc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd0a015e1e0 0x7fd0a006edc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0a00d3a00 msgr2=0x7fd0a00cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0a00d3a00 0x7fd0a00cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7fd0a5c5f700 1 -- 172.21.15.113:0/2322222668 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094009550 con 0x7fd0a00d0db0 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fd0a5c5f700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a00d0db0 0x7fd0a0171c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd09400b0e0 tx=0x7fd094004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd094003e40 con 0x7fd0a00d0db0 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7fe83ffff700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe840070fd0 0x7fe840152690 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe83000c7f0 tx=0x7fe83000cb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 -- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 msgr2=0x7f7f340ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7f3415d6a0 0x7f7f340ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 -- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f340b4090 msgr2=0x7f7f340ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f340b4090 0x7f7f340ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f7f32ffd700 1 -- 172.21.15.113:0/282956461 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c00c3d0 con 0x7f7f34071be0 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f7f32ffd700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f34071be0 0x7f7f3413ab20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7f34072f40 tx=0x7f7f2c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c01c040 con 0x7f7f34071be0 2024-01-16T13:52:10.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c0092e0 con 0x7f7f34071be0 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c004250 con 0x7f7f34071be0 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3413e940 con 0x7f7f34071be0 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3413ef00 con 0x7f7f34071be0 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7f14000fc0 con 0x7f7f34071be0 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e64167790 con 0x7f4e6415f460 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.794+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e640b3970 con 0x7f4e6415f460 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.796+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4e5c007230 con 0x7f4e6415f460 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f4e52ffd700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4e4406b420 0x7f4e4406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e5c0899d0 con 0x7f4e6415f460 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f4e68b85700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] conn(0x7f4e480015e0 0x7f4e48003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 --> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e48006bd0 con 0x7f4e480015e0 2024-01-16T13:52:10.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.797+0000 7f4e61d9b700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] conn(0x7f4e480015e0 0x7f4e48003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09400b660 con 0x7fd0a00d0db0 2024-01-16T13:52:10.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a00cd620 con 0x7fd0a00d0db0 2024-01-16T13:52:10.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09400ba10 con 0x7fd0a00d0db0 2024-01-16T13:52:10.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a00cbae0 con 0x7fd0a00d0db0 2024-01-16T13:52:10.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f2bf1af7700 1 Processor -- start 2024-01-16T13:52:10.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 -- 172.21.15.113:0/3827002357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a00b4260 msgr2=0x7f17a01e5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f17a00b4260 0x7f17a01e5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 -- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17a015d6a0 msgr2=0x7f17a01e3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17a015d6a0 0x7f17a01e3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 -- 172.21.15.113:0/3827002357 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179800c050 con 0x7f17a0071a10 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7ff3286e6700 1 -- 172.21.15.113:0/807580988 shutdown_connections 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830011800 con 0x7fe840070fd0 2024-01-16T13:52:10.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe830011960 con 0x7fe840070fd0 2024-01-16T13:52:10.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830011cf0 con 0x7fe840070fd0 2024-01-16T13:52:10.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe840164660 con 0x7fe840070fd0 2024-01-16T13:52:10.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe840160080 con 0x7fe840070fd0 2024-01-16T13:52:10.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f17a4868700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a0071a10 0x7f17a01f6500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f179400b410 tx=0x7f1794009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7f2c0043b0 con 0x7f7f34071be0 2024-01-16T13:52:10.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.798+0000 7f4e61d9b700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] conn(0x7f4e480015e0 0x7f4e48003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f4e6259c700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4e4406b420 0x7f4e4406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7ff3286e6700 1 -- 172.21.15.113:0/807580988 wait complete. 2024-01-16T13:52:10.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd09400bb70 con 0x7fd0a00d0db0 2024-01-16T13:52:10.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f7f1bfff700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f2c014070 con 0x7f7f34071be0 2024-01-16T13:52:10.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f7f1bfff700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] conn(0x7f7f200734f0 0x7f7f20075910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 --> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f20075fc0 con 0x7f7f200734f0 2024-01-16T13:52:10.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== osd.6 v2:172.21.15.124:6808/3167236260 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4e48006bd0 con 0x7f4e480015e0 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f7f337fe700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] conn(0x7f7f200734f0 0x7f7f20075910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.801+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f2c0553b0 con 0x7f7f34071be0 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.801+0000 7fd0927fc700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd08406b350 0x7fd08406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.801+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd09408afc0 con 0x7fd0a00d0db0 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794011830 con 0x7f17a0071a10 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17940119b0 con 0x7f17a0071a10 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794011d60 con 0x7f17a0071a10 2024-01-16T13:52:10.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a01e62b0 con 0x7f17a0071a10 2024-01-16T13:52:10.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a01f6a40 con 0x7f17a0071a10 2024-01-16T13:52:10.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.801+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f17a0072980 con 0x7f17a0071a10 2024-01-16T13:52:10.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7ff3286e6700 1 Processor -- start 2024-01-16T13:52:10.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe820000fc0 con 0x7fe840070fd0 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.799+0000 7f2bf1af7700 1 -- start start 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f2bf1af7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bec0b7430 0x7f2bec0cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f2bf1af7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bec0b9bf0 0x7f2bec070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f2bf1af7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0d3080 0x7f2bec0706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f2bf1af7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0c5450 con 0x7f2bec0d3080 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f2bf1af7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0c64d0 con 0x7f2bec0b9bf0 2024-01-16T13:52:10.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.800+0000 7f2bf1af7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0c5710 con 0x7f2bec0b7430 2024-01-16T13:52:10.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7fd0a545e700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd08406b350 0x7fd08406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f7f327fc700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f7f337fe700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] conn(0x7f7f200734f0 0x7f7f20075910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.801+0000 7f2beaffd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bec0b9bf0 0x7f2bec070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bec0b7430 0x7f2bec0cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bec0b7430 0x7f2bec0cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36728/0 (socket says 172.21.15.113:36728) 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 -- 172.21.15.113:0/4142378976 learned_addr learned my addr 172.21.15.113:0/4142378976 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2bebfff700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0d3080 0x7f2bec0706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 -- 172.21.15.113:0/4142378976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bec0b9bf0 msgr2=0x7f2bec070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bec0b9bf0 0x7f2bec070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 -- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0d3080 msgr2=0x7f2bec0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bec0d3080 0x7f2bec0706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f4e6259c700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4e4406b420 0x7f4e4406d8e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009a10 tx=0x7f4e58009190 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f179401a400 con 0x7f17a0071a10 2024-01-16T13:52:10.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7f2beb7fe700 1 -- 172.21.15.113:0/4142378976 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4009a70 con 0x7f2bec0b7430 2024-01-16T13:52:10.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.802+0000 7ff3286e6700 1 -- start start 2024-01-16T13:52:10.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== osd.5 v2:172.21.15.124:6824/23742019 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7f20075fc0 con 0x7f7f200734f0 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f179d7fa700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f178406b420 0x7f178406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f179408a7d0 con 0x7f17a0071a10 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f179d7fa700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] conn(0x7f1784073640 0x7f1784075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 --> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1784076110 con 0x7f1784073640 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f179408ab80 con 0x7f17a0071a10 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7ff3286e6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff32006f360 0x7ff3201ebda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff3286e6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3200ae4e0 0x7ff3200d0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff3286e6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff320162240 0x7ff320160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff3286e6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff320153550 con 0x7ff3200ae4e0 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff3286e6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3201613b0 con 0x7ff32006f360 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff3286e6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3200ce010 con 0x7ff320162240 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f2beb7fe700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bec0b7430 0x7f2bec0cd3e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b470 tx=0x7f2bdc00b830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc011830 con 0x7f2bec0b7430 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc011990 con 0x7f2bec0b7430 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc011d40 con 0x7f2bec0b7430 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec0c59f0 con 0x7f2bec0b7430 2024-01-16T13:52:10.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec0c5fb0 con 0x7f2bec0b7430 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] conn(0x7fd0a0153300 0x7fd0a006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 --> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0a00d2c30 con 0x7fd0a0153300 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fd0a545e700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd08406b350 0x7fd08406d810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0153800 tx=0x7fd09c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fd0a6460700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] conn(0x7fd0a0153300 0x7fd0a006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7fd0a6460700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] conn(0x7fd0a0153300 0x7fd0a006f300 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe830018360 con 0x7fe840070fd0 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fe83cff9700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe82c06b420 0x7fe82c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe830089a80 con 0x7fe840070fd0 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7fe83cff9700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] conn(0x7fe82c073640 0x7fe82c075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 --> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe82c076110 con 0x7fe82c073640 2024-01-16T13:52:10.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe83008e2a0 con 0x7fe840070fd0 2024-01-16T13:52:10.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.803+0000 7f7f327fc700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f341583c0 tx=0x7f7f28006cd0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7f179ffff700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f178406b420 0x7f178406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7f17a5069700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] conn(0x7f1784073640 0x7f1784075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7f179ffff700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f178406b420 0x7f178406d8e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f179800c2e0 tx=0x7f179800a000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7f17a5069700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] conn(0x7f1784073640 0x7f1784075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff326482700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff32006f360 0x7ff3201ebda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff326482700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff32006f360 0x7ff3201ebda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57728/0 (socket says 172.21.15.113:57728) 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7ff326482700 1 -- 172.21.15.113:0/2271712902 learned_addr learned my addr 172.21.15.113:0/2271712902 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326c83700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff320162240 0x7ff320160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326482700 1 -- 172.21.15.113:0/2271712902 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff320162240 msgr2=0x7ff320160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326482700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff320162240 0x7ff320160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326482700 1 -- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3200ae4e0 msgr2=0x7ff3200d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326482700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3200ae4e0 0x7ff3200d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326482700 1 -- 172.21.15.113:0/2271712902 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c009550 con 0x7ff32006f360 2024-01-16T13:52:10.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2bec0b3670 con 0x7f2bec0b7430 2024-01-16T13:52:10.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7fe8448eb700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] conn(0x7fe82c073640 0x7fe82c075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.804+0000 7fe83f7fe700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe82c06b420 0x7fe82c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7fe8448eb700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] conn(0x7fe82c073640 0x7fe82c075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7fe83f7fe700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe82c06b420 0x7fe82c06d8e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8401fb530 tx=0x7fe834008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2bdc01a400 con 0x7f2bec0b7430 2024-01-16T13:52:10.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.805+0000 7ff326482700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff32006f360 0x7ff3201ebda0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff3200728e0 tx=0x7ff31c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0036a0 con 0x7ff32006f360 2024-01-16T13:52:10.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3200ce2b0 con 0x7ff32006f360 2024-01-16T13:52:10.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3200ce870 con 0x7ff32006f360 2024-01-16T13:52:10.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c003820 con 0x7ff32006f360 2024-01-16T13:52:10.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c003bd0 con 0x7ff32006f360 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2bd806b350 0x7f2bd806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bdc089ba0 con 0x7f2bec0b7430 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2beaffd700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2bd806b350 0x7f2bd806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2bd3fff700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] conn(0x7f2bd80734f0 0x7f2bd8075910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 --> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2bd8075fc0 con 0x7f2bd80734f0 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== osd.3 v2:172.21.15.113:6800/3595238614 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd098001800 con 0x7fd0a0153300 2024-01-16T13:52:10.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.807+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff308000fc0 con 0x7ff32006f360 2024-01-16T13:52:10.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bdc011ea0 con 0x7f2bec0b7430 2024-01-16T13:52:10.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7f2bebfff700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] conn(0x7f2bd80734f0 0x7f2bd8075910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.806+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== osd.4 v2:172.21.15.124:6800/2883525511 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe82c076110 con 0x7fe82c073640 2024-01-16T13:52:10.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.808+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff31c003d30 con 0x7ff32006f360 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.809+0000 7ff316ffd700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff30406b350 0x7ff30406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.809+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff31c08b200 con 0x7ff32006f360 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.809+0000 7ff316ffd700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] conn(0x7ff3040734f0 0x7ff304075910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.809+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 --> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff304075fc0 con 0x7ff3040734f0 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.807+0000 7f2bebfff700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] conn(0x7f2bd80734f0 0x7f2bd8075910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.807+0000 7f2beaffd700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2bd806b350 0x7f2bd806d810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2be0007c20 tx=0x7f2be0007d70 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.809+0000 7ff326c83700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] conn(0x7ff3040734f0 0x7ff304075910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== osd.0 v2:172.21.15.113:6816/1924924117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1784076110 con 0x7f1784073640 2024-01-16T13:52:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7ff325c81700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff30406b350 0x7ff30406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7ff326c83700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] conn(0x7ff3040734f0 0x7ff304075910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7ff325c81700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff30406b350 0x7ff30406d810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff3201653d0 tx=0x7ff310007ae0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff31c056060 con 0x7ff32006f360 2024-01-16T13:52:10.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== osd.2 v2:172.21.15.113:6824/1336088369 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2bd8075fc0 con 0x7f2bd80734f0 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7ff1e5442700 1 Processor -- start 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.810+0000 7ff1e5442700 1 -- start start 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.811+0000 7ff1e5442700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff1e015d6a0 0x7ff1e00b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.811+0000 7ff1e5442700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff1e00b4090 0x7ff1e00714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.811+0000 7ff1e5442700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e0071a10 0x7ff1e006cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.811+0000 7ff1e5442700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e0072d60 con 0x7ff1e015d6a0 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.811+0000 7ff1e5442700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e006de80 con 0x7ff1e00b4090 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.811+0000 7ff1e5442700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e00b4510 con 0x7ff1e0071a10 2024-01-16T13:52:10.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff1e015d6a0 0x7ff1e00b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff1e015d6a0 0x7ff1e00b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40504/0 (socket says 172.21.15.113:40504) 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 -- 172.21.15.113:0/1686619359 learned_addr learned my addr 172.21.15.113:0/1686619359 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1de7fc700 1 --2- 172.21.15.113:0/1686619359 >> v2:172.21.15.124:3300/0 conn(0x7ff1e00b4090 0x7ff1e00714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1df7fe700 1 --2- 172.21.15.113:0/1686619359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e0071a10 0x7ff1e006cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 -- 172.21.15.113:0/1686619359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e0071a10 msgr2=0x7ff1e006cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 --2- 172.21.15.113:0/1686619359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e0071a10 0x7ff1e006cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 -- 172.21.15.113:0/1686619359 >> v2:172.21.15.124:3300/0 conn(0x7ff1e00b4090 msgr2=0x7ff1e00714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 --2- 172.21.15.113:0/1686619359 >> v2:172.21.15.124:3300/0 conn(0x7ff1e00b4090 0x7ff1e00714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.812+0000 7ff1deffd700 1 -- 172.21.15.113:0/1686619359 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00720e0 con 0x7ff1e015d6a0 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.813+0000 7ff1deffd700 1 --2- 172.21.15.113:0/1686619359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e015d6a0 0x7ff1e00b3b50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0062350 tx=0x7ff1d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a381a022987dd4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.814+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== osd.1 v2:172.21.15.113:6808/1956545854 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff304075fc0 con 0x7ff3040734f0 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.813+0000 7ff1ddffb700 1 -- 172.21.15.113:0/1686619359 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800e070 con 0x7ff1e015d6a0 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.813+0000 7ff1ddffb700 1 -- 172.21.15.113:0/1686619359 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d800bb80 con 0x7ff1e015d6a0 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.813+0000 7ff1ddffb700 1 -- 172.21.15.113:0/1686619359 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80126e0 con 0x7ff1e015d6a0 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 -- 172.21.15.113:0/1686619359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e015d6a0 msgr2=0x7ff1e00b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1686619359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e015d6a0 0x7ff1e00b3b50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0062350 tx=0x7ff1d800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 -- 172.21.15.113:0/1686619359 shutdown_connections 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1686619359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e015d6a0 0x7ff1e00b3b50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1686619359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e0071a10 0x7ff1e006cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1686619359 >> v2:172.21.15.124:3300/0 conn(0x7ff1e00b4090 0x7ff1e00714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 -- 172.21.15.113:0/1686619359 >> 172.21.15.113:0/1686619359 conn(0x7ff1e00cfa30 msgr2=0x7ff1e00cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 -- 172.21.15.113:0/1686619359 shutdown_connections 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.815+0000 7ff1e5442700 1 -- 172.21.15.113:0/1686619359 wait complete. 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.817+0000 7ff1e5442700 1 Processor -- start 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.817+0000 7ff1e5442700 1 -- start start 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1e5442700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e0071a10 0x7ff1e00aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1e5442700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1e00b4090 0x7ff1e00ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1e5442700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e015d6a0 0x7ff1e02002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1e5442700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e00aa3d0 con 0x7ff1e0071a10 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1e5442700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e00a97e0 con 0x7ff1e00b4090 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1e5442700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e00abe10 con 0x7ff1e015d6a0 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e0071a10 0x7ff1e00aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e0071a10 0x7ff1e00aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40510/0 (socket says 172.21.15.113:40510) 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.819+0000 7ff1deffd700 1 -- 172.21.15.113:0/1286559408 learned_addr learned my addr 172.21.15.113:0/1286559408 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.820+0000 7ff1deffd700 1 -- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e015d6a0 msgr2=0x7ff1e02002b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.820+0000 7ff1deffd700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1e015d6a0 0x7ff1e02002b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.820+0000 7ff1deffd700 1 -- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1e00b4090 msgr2=0x7ff1e00ab8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.820+0000 7ff1deffd700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1e00b4090 0x7ff1e00ab8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.820+0000 7ff1deffd700 1 -- 172.21.15.113:0/1286559408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d8009b30 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.820+0000 7ff1deffd700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e0071a10 0x7ff1e00aef50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00737d0 tx=0x7ff1d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.823+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80092e0 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.823+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d8003c30 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.823+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8004050 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.823+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e00a9a30 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.823+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e00a9ff0 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.826+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff1d80041b0 con 0x7ff1e0071a10 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.827+0000 7ff1c77fe700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff1cc06b420 0x7ff1cc06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.827+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1d808b070 con 0x7ff1e0071a10 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.830+0000 7ff1de7fc700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff1cc06b420 0x7ff1cc06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.830+0000 7ff1de7fc700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff1cc06b420 0x7ff1cc06d8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1e015afc0 tx=0x7ff1d0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.831+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] conn(0x7ff1c00015e0 0x7ff1c0003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.831+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 --> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1c0006bd0 con 0x7ff1c00015e0 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.832+0000 7ff1df7fe700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] conn(0x7ff1c00015e0 0x7ff1c0003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.831+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 --> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e48005c50 con 0x7f4e480015e0 2024-01-16T13:52:10.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.832+0000 7ff1df7fe700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] conn(0x7ff1c00015e0 0x7ff1c0003aa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:10.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.833+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== osd.7 v2:172.21.15.124:6809/2264364451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff1c0006bd0 con 0x7ff1c00015e0 2024-01-16T13:52:10.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.833+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 --> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe820002d40 con 0x7fe82c073640 2024-01-16T13:52:10.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.834+0000 7fe83cff9700 1 -- 172.21.15.113:0/2098609986 <== osd.4 v2:172.21.15.124:6800/2883525511 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe820002d40 con 0x7fe82c073640 2024-01-16T13:52:10.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.835+0000 7f4e52ffd700 1 -- 172.21.15.113:0/2103859906 <== osd.6 v2:172.21.15.124:6808/3167236260 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e48005c50 con 0x7f4e480015e0 2024-01-16T13:52:10.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] conn(0x7f4e480015e0 msgr2=0x7f4e48003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:6808/3167236260,v1:172.21.15.124:6810/3167236260] conn(0x7f4e480015e0 0x7f4e48003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4e4406b420 msgr2=0x7f4e4406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4e4406b420 0x7f4e4406d8e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009a10 tx=0x7f4e58009190 comp rx=0 tx=0).stop 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4e6415f460 msgr2=0x7f4e6413d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 --2- 172.21.15.113:0/2103859906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4e6415f460 0x7f4e6413d0b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00cbd0 tx=0x7f4e5c00cee0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e62d9d700 1 -- 172.21.15.113:0/2103859906 reap_dead start 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 shutdown_connections 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 >> 172.21.15.113:0/2103859906 conn(0x7f4e640cd570 msgr2=0x7f4e6406f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 shutdown_connections 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.836+0000 7f4e68b85700 1 -- 172.21.15.113:0/2103859906 wait complete. 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.838+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 --> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0a00cb940 con 0x7fd0a0153300 2024-01-16T13:52:10.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.840+0000 7fd0927fc700 1 -- 172.21.15.113:0/2322222668 <== osd.3 v2:172.21.15.113:6800/3595238614 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0a00cb940 con 0x7fd0a0153300 2024-01-16T13:52:10.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.840+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 --> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7f14002d40 con 0x7f7f200734f0 2024-01-16T13:52:10.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.841+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 --> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2bec163060 con 0x7f2bd80734f0 2024-01-16T13:52:10.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.841+0000 7f7f1bfff700 1 -- 172.21.15.113:0/282956461 <== osd.5 v2:172.21.15.124:6824/23742019 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7f14002d40 con 0x7f7f200734f0 2024-01-16T13:52:10.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] conn(0x7fe82c073640 msgr2=0x7fe82c075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:6800/2883525511,v1:172.21.15.124:6801/2883525511] conn(0x7fe82c073640 0x7fe82c075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe82c06b420 msgr2=0x7fe82c06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe82c06b420 0x7fe82c06d8e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8401fb530 tx=0x7fe834008030 comp rx=0 tx=0).stop 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe840070fd0 msgr2=0x7fe840152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 --2- 172.21.15.113:0/2098609986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe840070fd0 0x7fe840152690 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe83000c7f0 tx=0x7fe83000cb00 comp rx=0 tx=0).stop 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe8448eb700 1 -- 172.21.15.113:0/2098609986 reap_dead start 2024-01-16T13:52:10.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 --> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f17a00726a0 con 0x7f1784073640 2024-01-16T13:52:10.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 shutdown_connections 2024-01-16T13:52:10.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 >> 172.21.15.113:0/2098609986 conn(0x7fe8400b1740 msgr2=0x7fe8400b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 shutdown_connections 2024-01-16T13:52:10.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.842+0000 7fe84634e700 1 -- 172.21.15.113:0/2098609986 wait complete. 2024-01-16T13:52:10.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] conn(0x7fd0a0153300 msgr2=0x7fd0a006f300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6800/3595238614,v1:172.21.15.113:6801/3595238614] conn(0x7fd0a0153300 0x7fd0a006f300 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7f2bd3fff700 1 -- 172.21.15.113:0/4142378976 <== osd.2 v2:172.21.15.113:6824/1336088369 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2be4003c20 con 0x7f2bd80734f0 2024-01-16T13:52:10.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd08406b350 msgr2=0x7fd08406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd08406b350 0x7fd08406d810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0153800 tx=0x7fd09c008040 comp rx=0 tx=0).stop 2024-01-16T13:52:10.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a00d0db0 msgr2=0x7fd0a0171c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 --2- 172.21.15.113:0/2322222668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd0a00d0db0 0x7fd0a0171c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd09400b0e0 tx=0x7fd094004750 comp rx=0 tx=0).stop 2024-01-16T13:52:10.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 --> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff308002d40 con 0x7ff3040734f0 2024-01-16T13:52:10.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a6460700 1 -- 172.21.15.113:0/2322222668 reap_dead start 2024-01-16T13:52:10.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 shutdown_connections 2024-01-16T13:52:10.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.844+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 >> 172.21.15.113:0/2322222668 conn(0x7fd0a00b5b50 msgr2=0x7fd0a00b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 shutdown_connections 2024-01-16T13:52:10.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7fd0a7ec3700 1 -- 172.21.15.113:0/2322222668 wait complete. 2024-01-16T13:52:10.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff316ffd700 1 -- 172.21.15.113:0/2271712902 <== osd.1 v2:172.21.15.113:6808/1956545854 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff308002d40 con 0x7ff3040734f0 2024-01-16T13:52:10.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] conn(0x7ff3040734f0 msgr2=0x7ff304075910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff3286e6700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6808/1956545854,v1:172.21.15.113:6809/1956545854] conn(0x7ff3040734f0 0x7ff304075910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff30406b350 msgr2=0x7ff30406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff3286e6700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff30406b350 0x7ff30406d810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff3201653d0 tx=0x7ff310007ae0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff32006f360 msgr2=0x7ff3201ebda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff3286e6700 1 --2- 172.21.15.113:0/2271712902 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff32006f360 0x7ff3201ebda0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff3200728e0 tx=0x7ff31c004750 comp rx=0 tx=0).stop 2024-01-16T13:52:10.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.845+0000 7ff326c83700 1 -- 172.21.15.113:0/2271712902 reap_dead start 2024-01-16T13:52:10.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f179d7fa700 1 -- 172.21.15.113:0/3827002357 <== osd.0 v2:172.21.15.113:6816/1924924117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1790009960 con 0x7f1784073640 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] conn(0x7f1784073640 msgr2=0x7f1784075a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6816/1924924117,v1:172.21.15.113:6817/1924924117] conn(0x7f1784073640 0x7f1784075a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f178406b420 msgr2=0x7f178406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f178406b420 0x7f178406d8e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f179800c2e0 tx=0x7f179800a000 comp rx=0 tx=0).stop 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a0071a10 msgr2=0x7f17a01f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 --2- 172.21.15.113:0/3827002357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17a0071a10 0x7f17a01f6500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f179400b410 tx=0x7f1794009f90 comp rx=0 tx=0).stop 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a5069700 1 -- 172.21.15.113:0/3827002357 reap_dead start 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 shutdown_connections 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.846+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 >> 172.21.15.113:0/3827002357 conn(0x7f17a00cfa30 msgr2=0x7f17a00af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.849+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 shutdown_connections 2024-01-16T13:52:10.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.849+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 >> 172.21.15.113:0/2271712902 conn(0x7ff320071b90 msgr2=0x7ff320073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.849+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 shutdown_connections 2024-01-16T13:52:10.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.849+0000 7ff3286e6700 1 -- 172.21.15.113:0/2271712902 wait complete. 2024-01-16T13:52:10.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.852+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 shutdown_connections 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.852+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 --> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff1c0005cb0 con 0x7ff1c00015e0 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.852+0000 7f17a6acc700 1 -- 172.21.15.113:0/3827002357 wait complete. 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.852+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] conn(0x7f2bd80734f0 msgr2=0x7f2bd8075910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.852+0000 7f2bf1af7700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6824/1336088369,v1:172.21.15.113:6825/1336088369] conn(0x7f2bd80734f0 0x7f2bd8075910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2bd806b350 msgr2=0x7f2bd806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2bd806b350 0x7f2bd806d810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2be0007c20 tx=0x7f2be0007d70 comp rx=0 tx=0).stop 2024-01-16T13:52:10.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bec0b7430 msgr2=0x7f2bec0cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 --2- 172.21.15.113:0/4142378976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bec0b7430 0x7f2bec0cd3e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b470 tx=0x7f2bdc00b830 comp rx=0 tx=0).stop 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] conn(0x7f7f200734f0 msgr2=0x7f7f20075910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:6824/23742019,v1:172.21.15.124:6825/23742019] conn(0x7f7f200734f0 0x7f7f20075910 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 msgr2=0x7f7f2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7f2006b350 0x7f7f2006d810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f341583c0 tx=0x7f7f28006cd0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f34071be0 msgr2=0x7f7f3413ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 --2- 172.21.15.113:0/282956461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7f34071be0 0x7f7f3413ab20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7f34072f40 tx=0x7f7f2c004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f337fe700 1 -- 172.21.15.113:0/282956461 reap_dead start 2024-01-16T13:52:10.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bebfff700 1 -- 172.21.15.113:0/4142378976 reap_dead start 2024-01-16T13:52:10.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 shutdown_connections 2024-01-16T13:52:10.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 >> 172.21.15.113:0/4142378976 conn(0x7f2bec0d3ed0 msgr2=0x7f2bec0b61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7ff1c77fe700 1 -- 172.21.15.113:0/1286559408 <== osd.7 v2:172.21.15.124:6809/2264364451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff1c0005cb0 con 0x7ff1c00015e0 2024-01-16T13:52:10.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 shutdown_connections 2024-01-16T13:52:10.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f2bf1af7700 1 -- 172.21.15.113:0/4142378976 wait complete. 2024-01-16T13:52:10.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 shutdown_connections 2024-01-16T13:52:10.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.853+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 >> 172.21.15.113:0/282956461 conn(0x7f7f340cfa30 msgr2=0x7f7f34143660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] conn(0x7ff1c00015e0 msgr2=0x7ff1c0003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.124:6809/2264364451,v1:172.21.15.124:6811/2264364451] conn(0x7ff1c00015e0 0x7ff1c0003aa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff1cc06b420 msgr2=0x7ff1cc06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff1cc06b420 0x7ff1cc06d8e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1e015afc0 tx=0x7ff1d0008040 comp rx=0 tx=0).stop 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e0071a10 msgr2=0x7ff1e00aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 --2- 172.21.15.113:0/1286559408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1e0071a10 0x7ff1e00aef50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00737d0 tx=0x7ff1d8004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1df7fe700 1 -- 172.21.15.113:0/1286559408 reap_dead start 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 shutdown_connections 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 >> 172.21.15.113:0/1286559408 conn(0x7ff1e00cfa30 msgr2=0x7ff1e0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 shutdown_connections 2024-01-16T13:52:10.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7f7f398aa700 1 -- 172.21.15.113:0/282956461 wait complete. 2024-01-16T13:52:10.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 shutdown_connections 2024-01-16T13:52:10.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:10.854+0000 7ff1e5442700 1 -- 172.21.15.113:0/1286559408 wait complete. 2024-01-16T13:52:10.852 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:10.853 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T13:52:10.854 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:10.854 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T13:52:10.855 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-16T13:52:10.855 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T13:52:10.856 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-16T13:52:10.857 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T13:52:10.858 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:10.859 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T13:52:10.860 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:10.861 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T13:52:10.862 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:10.862 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:10.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T13:52:10.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T13:52:11.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.053+0000 7f9ca378f700 1 Processor -- start 2024-01-16T13:52:11.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.054+0000 7f9ca378f700 1 -- start start 2024-01-16T13:52:11.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.055+0000 7f70cb627700 1 Processor -- start 2024-01-16T13:52:11.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.055+0000 7f9ca378f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.055+0000 7f9ca378f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9c9c0b4090 0x7f9c9c0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.055+0000 7f9ca378f700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c071e70 con 0x7f9c9c071a10 2024-01-16T13:52:11.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.055+0000 7f9ca378f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c072d60 con 0x7f9c9c0b4090 2024-01-16T13:52:11.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.055+0000 7f9ca378f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c06de80 con 0x7f9c9c15d6a0 2024-01-16T13:52:11.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca0d2a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9c9c0b4090 0x7f9c9c0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca0d2a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9c9c0b4090 0x7f9c9c0714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57742/0 (socket says 172.21.15.113:57742) 2024-01-16T13:52:11.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca0d2a700 1 -- 172.21.15.113:0/2048898700 learned_addr learned my addr 172.21.15.113:0/2048898700 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca152b700 1 --2- 172.21.15.113:0/2048898700 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca152b700 1 -- 172.21.15.113:0/2048898700 >> v2:172.21.15.124:3300/0 conn(0x7f9c9c0b4090 msgr2=0x7f9c9c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca152b700 1 --2- 172.21.15.113:0/2048898700 >> v2:172.21.15.124:3300/0 conn(0x7f9c9c0b4090 0x7f9c9c0714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca152b700 1 -- 172.21.15.113:0/2048898700 >> v1:172.21.15.124:6789/0 conn(0x7f9c9c071a10 legacy=0x7f9c9c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f9ca152b700 1 -- 172.21.15.113:0/2048898700 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c0720e0 con 0x7f9c9c15d6a0 2024-01-16T13:52:11.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f9ca152b700 1 --2- 172.21.15.113:0/2048898700 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c0b3b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009840 tx=0x7f9c98009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=769a7e1ad73206b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.056+0000 7f70cb627700 1 -- start start 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70cb627700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f70c415af10 0x7f70c415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70cb627700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c40ce140 0x7f70c415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70cb627700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c41414c0 con 0x7f70c415f460 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70cb627700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c415f8c0 con 0x7f70c415af10 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70cb627700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40ce9f0 con 0x7f70c40ce140 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70c93c3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f70c415af10 0x7f70c415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70c93c3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f70c415af10 0x7f70c415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40526/0 (socket says 172.21.15.113:40526) 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f70c93c3700 1 -- 172.21.15.113:0/819456167 learned_addr learned my addr 172.21.15.113:0/819456167 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70c8bc2700 1 --2- 172.21.15.113:0/819456167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c40ce140 0x7f70c415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70bbfff700 1 -- 172.21.15.113:0/819456167 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782886880 0 0) 0x7f70c41414c0 con 0x7f70c415f460 2024-01-16T13:52:11.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70bbfff700 1 -- 172.21.15.113:0/819456167 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b00035b0 con 0x7f70c415f460 2024-01-16T13:52:11.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f9c93fff700 1 -- 172.21.15.113:0/2048898700 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98011830 con 0x7f9c9c15d6a0 2024-01-16T13:52:11.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f9c93fff700 1 -- 172.21.15.113:0/2048898700 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98011990 con 0x7f9c9c15d6a0 2024-01-16T13:52:11.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.057+0000 7f9c93fff700 1 -- 172.21.15.113:0/2048898700 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98011d00 con 0x7f9c9c15d6a0 2024-01-16T13:52:11.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 -- 172.21.15.113:0/2048898700 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 msgr2=0x7f9c9c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2048898700 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c0b3b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009840 tx=0x7f9c98009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70c93c3700 1 -- 172.21.15.113:0/819456167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c40ce140 msgr2=0x7f70c415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70c93c3700 1 --2- 172.21.15.113:0/819456167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c40ce140 0x7f70c415ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70c93c3700 1 -- 172.21.15.113:0/819456167 >> v1:172.21.15.124:6789/0 conn(0x7f70c415f460 legacy=0x7f70c415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70c93c3700 1 -- 172.21.15.113:0/819456167 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c40cecd0 con 0x7f70c415af10 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 -- 172.21.15.113:0/2048898700 shutdown_connections 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2048898700 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c0b3b50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2048898700 >> v2:172.21.15.124:3300/0 conn(0x7f9c9c0b4090 0x7f9c9c0714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 -- 172.21.15.113:0/2048898700 >> 172.21.15.113:0/2048898700 conn(0x7f9c9c0cfa30 msgr2=0x7f9c9c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 -- 172.21.15.113:0/2048898700 shutdown_connections 2024-01-16T13:52:11.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f70c93c3700 1 --2- 172.21.15.113:0/819456167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c415af10 0x7f70c415e9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00d150 tx=0x7f70bc00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225ec909f0469fcc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.059+0000 7f70bbfff700 1 -- 172.21.15.113:0/819456167 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc010070 con 0x7f70c415af10 2024-01-16T13:52:11.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.059+0000 7f70bbfff700 1 -- 172.21.15.113:0/819456167 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc00dcd0 con 0x7f70c415af10 2024-01-16T13:52:11.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.059+0000 7f70bbfff700 1 -- 172.21.15.113:0/819456167 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc00b310 con 0x7f70c415af10 2024-01-16T13:52:11.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.060+0000 7f70cb627700 1 -- 172.21.15.113:0/819456167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c415af10 msgr2=0x7f70c415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.060+0000 7f70cb627700 1 --2- 172.21.15.113:0/819456167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c415af10 0x7f70c415e9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00d150 tx=0x7f70bc00d460 comp rx=0 tx=0).stop 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.058+0000 7f9ca378f700 1 -- 172.21.15.113:0/2048898700 wait complete. 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.060+0000 7f9ca378f700 1 Processor -- start 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.060+0000 7f9ca378f700 1 -- start start 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca378f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 0x7f9c9c1393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca378f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c9c0b4090 0x7f9c9c156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca378f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca378f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c157a30 con 0x7f9c9c0b4090 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca378f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c157d10 con 0x7f9c9c071a10 2024-01-16T13:52:11.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca378f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c159f20 con 0x7f9c9c15d6a0 2024-01-16T13:52:11.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca152b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 0x7f9c9c1393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca152b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 0x7f9c9c1393f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57756/0 (socket says 172.21.15.113:57756) 2024-01-16T13:52:11.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca152b700 1 -- 172.21.15.113:0/2296191464 learned_addr learned my addr 172.21.15.113:0/2296191464 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f9ca1d2c700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f9ca152b700 1 -- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 msgr2=0x7f9c9c158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f9ca152b700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f9ca152b700 1 -- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c9c0b4090 msgr2=0x7f9c9c156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f9ca152b700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c9c0b4090 0x7f9c9c156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f9ca152b700 1 -- 172.21.15.113:0/2296191464 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c980094a0 con 0x7f9c9c071a10 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f70cb627700 1 -- 172.21.15.113:0/819456167 shutdown_connections 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f70cb627700 1 --2- 172.21.15.113:0/819456167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c415af10 0x7f70c415e9e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f70cb627700 1 --2- 172.21.15.113:0/819456167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c40ce140 0x7f70c415ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f70cb627700 1 -- 172.21.15.113:0/819456167 >> 172.21.15.113:0/819456167 conn(0x7f70c40cd570 msgr2=0x7f70c40cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f9ca152b700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 0x7f9c9c1393f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9c98014040 tx=0x7f9c98004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.060+0000 7f46c2be5700 1 Processor -- start 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.060+0000 7f46c2be5700 1 -- start start 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46c2be5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f46bc0b3d40 0x7f46bc0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46c2be5700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc06d550 con 0x7f46bc070d60 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46c2be5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46bc06cc60 con 0x7f46bc140680 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46c2be5700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc0b47c0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46c1182700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f46bc070d60 0x7f46bc06d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:52054/0 (socket says 172.21.15.113:52054) 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46c1182700 1 -- 172.21.15.113:0/310265395 learned_addr learned my addr 172.21.15.113:0/310265395 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.062+0000 7f46bbfff700 1 --2- 172.21.15.113:0/310265395 >> v2:172.21.15.113:3300/0 conn(0x7f46bc0b3d40 0x7f46bc0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f46bbfff700 1 -- 172.21.15.113:0/310265395 >> v1:172.21.15.124:6789/0 conn(0x7f46bc140680 legacy=0x7f46bc140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f46bbfff700 1 -- 172.21.15.113:0/310265395 >> v1:172.21.15.113:6789/0 conn(0x7f46bc070d60 legacy=0x7f46bc06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f46bbfff700 1 -- 172.21.15.113:0/310265395 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc06cf00 con 0x7f46bc0b3d40 2024-01-16T13:52:11.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f70cb627700 1 -- 172.21.15.113:0/819456167 shutdown_connections 2024-01-16T13:52:11.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f46bbfff700 1 --2- 172.21.15.113:0/310265395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc0b41d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009870 tx=0x7f46ac009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1937065c74f813 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.064+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c980036a0 con 0x7f9c9c071a10 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.064+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9801f070 con 0x7f9c9c071a10 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.064+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98003a00 con 0x7f9c9c071a10 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.064+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c15a200 con 0x7f9c9c071a10 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.064+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c139930 con 0x7f9c9c071a10 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.063+0000 7f70cb627700 1 -- 172.21.15.113:0/819456167 wait complete. 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.065+0000 7f70cb627700 1 Processor -- start 2024-01-16T13:52:11.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.065+0000 7f70cb627700 1 -- start start 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.066+0000 7f46bb7fe700 1 -- 172.21.15.113:0/310265395 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac004230 con 0x7f46bc0b3d40 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.066+0000 7f46bb7fe700 1 -- 172.21.15.113:0/310265395 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac004390 con 0x7f46bc0b3d40 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.066+0000 7f46bb7fe700 1 -- 172.21.15.113:0/310265395 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac0036a0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70cb627700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 0x7f70c4141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70cb627700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70c415af10 0x7f70c41fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70cb627700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c415f460 0x7f70c41fbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70cb627700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40b36f0 con 0x7f70c40ce140 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70cb627700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40b39d0 con 0x7f70c415af10 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70cb627700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c413f870 con 0x7f70c415f460 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70c93c3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 0x7f70c4141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70c93c3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 0x7f70c4141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40550/0 (socket says 172.21.15.113:40550) 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70c93c3700 1 -- 172.21.15.113:0/3772403196 learned_addr learned my addr 172.21.15.113:0/3772403196 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70c8bc2700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70c415af10 0x7f70c41fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7c0052f0 con 0x7f9c9c071a10 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f46c2be5700 1 -- 172.21.15.113:0/310265395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 msgr2=0x7f46bc0b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f46c2be5700 1 --2- 172.21.15.113:0/310265395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc0b41d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009870 tx=0x7f46ac009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f70c9bc4700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c415f460 0x7f70c41fbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c93c3700 1 -- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c415f460 msgr2=0x7f70c41fbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c93c3700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c415f460 0x7f70c41fbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c93c3700 1 -- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70c415af10 msgr2=0x7f70c41fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c93c3700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70c415af10 0x7f70c41fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c93c3700 1 -- 172.21.15.113:0/3772403196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc00bd40 con 0x7f70c40ce140 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c9bc4700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c415f460 0x7f70c41fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:11.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c8bc2700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70c415af10 0x7f70c41fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:11.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f46c2be5700 1 -- 172.21.15.113:0/310265395 shutdown_connections 2024-01-16T13:52:11.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f46c2be5700 1 --2- 172.21.15.113:0/310265395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc0b41d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f46c2be5700 1 -- 172.21.15.113:0/310265395 >> 172.21.15.113:0/310265395 conn(0x7f46bc0b1740 msgr2=0x7f46bc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f46c2be5700 1 -- 172.21.15.113:0/310265395 shutdown_connections 2024-01-16T13:52:11.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f70c93c3700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 0x7f70c4141a70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70c4073510 tx=0x7f70bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f46c2be5700 1 -- 172.21.15.113:0/310265395 wait complete. 2024-01-16T13:52:11.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9c98003b60 con 0x7f9c9c071a10 2024-01-16T13:52:11.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f9c91ffb700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c8006b420 0x7f9c8006d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c98021070 con 0x7f9c9c071a10 2024-01-16T13:52:11.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc009600 con 0x7f70c40ce140 2024-01-16T13:52:11.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc009780 con 0x7f70c40ce140 2024-01-16T13:52:11.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc009b30 con 0x7f70c40ce140 2024-01-16T13:52:11.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c413fb50 con 0x7f70c40ce140 2024-01-16T13:52:11.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.069+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c4142190 con 0x7f70c40ce140 2024-01-16T13:52:11.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.070+0000 7f46c2be5700 1 Processor -- start 2024-01-16T13:52:11.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.071+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c413fce0 con 0x7f70c40ce140 2024-01-16T13:52:11.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.071+0000 7f9ca0d2a700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c8006b420 0x7f9c8006d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.071+0000 7f9ca0d2a700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c8006b420 0x7f9c8006d8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0085a0 tx=0x7f9c8c0084f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.070+0000 7f46c2be5700 1 -- start start 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46c2be5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46bc070d60 0x7f46bc0ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46c2be5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46c2be5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46bc140680 0x7f46bc15eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46c2be5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0a94d0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46c2be5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc15d970 con 0x7f46bc070d60 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46c2be5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc165ab0 con 0x7f46bc140680 2024-01-16T13:52:11.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46bbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46bbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc15d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40564/0 (socket says 172.21.15.113:40564) 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f46bbfff700 1 -- 172.21.15.113:0/4080227822 learned_addr learned my addr 172.21.15.113:0/4080227822 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46c0981700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46bc070d60 0x7f46bc0ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46c1182700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46bc140680 0x7f46bc15eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46bbfff700 1 -- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46bc140680 msgr2=0x7f46bc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46bbfff700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46bc140680 0x7f46bc15eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46bbfff700 1 -- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46bc070d60 msgr2=0x7f46bc0ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46bbfff700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46bc070d60 0x7f46bc0ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46bbfff700 1 -- 172.21.15.113:0/4080227822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac0094a0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f70bc009020 con 0x7f70c40ce140 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.072+0000 7f70b9ffb700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f70a806b420 0x7f70a806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70bc00de40 con 0x7f70c40ce140 2024-01-16T13:52:11.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f70c8bc2700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f70a806b420 0x7f70a806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.073+0000 7f46bbfff700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc15d430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0737d0 tx=0x7f46ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.074+0000 7f70c8bc2700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f70a806b420 0x7f70a806d8e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70c40b1360 tx=0x7f70b4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.065+0000 7f50821b7700 1 Processor -- start 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.065+0000 7f50821b7700 1 -- start start 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f50821b7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f507c16ae60 0x7f507c06efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f50821b7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c072bb0 0x7f507c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f50821b7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c08bdb0 con 0x7f507c0b4090 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f50821b7700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0b4510 con 0x7f507c16ae60 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f50821b7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c06ce30 con 0x7f507c072bb0 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f507b7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f507c16ae60 0x7f507c06efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f507b7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f507c16ae60 0x7f507c06efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40560/0 (socket says 172.21.15.113:40560) 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f507b7fe700 1 -- 172.21.15.113:0/1401494089 learned_addr learned my addr 172.21.15.113:0/1401494089 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.067+0000 7f507affd700 1 --2- 172.21.15.113:0/1401494089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c072bb0 0x7f507c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.068+0000 7f507a7fc700 1 -- 172.21.15.113:0/1401494089 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28431101 0 0) 0x7f507c08bdb0 con 0x7f507c0b4090 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.075+0000 7f507b7fe700 1 -- 172.21.15.113:0/1401494089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c072bb0 msgr2=0x7f507c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.075+0000 7f507b7fe700 1 --2- 172.21.15.113:0/1401494089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c072bb0 0x7f507c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.075+0000 7f507b7fe700 1 -- 172.21.15.113:0/1401494089 >> v1:172.21.15.113:6789/0 conn(0x7f507c0b4090 legacy=0x7f507c16e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.075+0000 7f507b7fe700 1 -- 172.21.15.113:0/1401494089 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c06d110 con 0x7f507c16ae60 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.075+0000 7f507b7fe700 1 --2- 172.21.15.113:0/1401494089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c16ae60 0x7f507c06efc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5074007eb0 tx=0x7f507400d480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fbcc242242c4d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.076+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c980561a0 con 0x7f9c9c071a10 2024-01-16T13:52:11.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac010590 con 0x7f46bc0b3d40 2024-01-16T13:52:11.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac010710 con 0x7f46bc0b3d40 2024-01-16T13:52:11.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac010ac0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc0aced0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc0ad490 con 0x7f46bc0b3d40 2024-01-16T13:52:11.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70bc0555d0 con 0x7f70c40ce140 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f507a7fc700 1 -- 172.21.15.113:0/1401494089 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074012440 con 0x7f507c16ae60 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f507a7fc700 1 -- 172.21.15.113:0/1401494089 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50740125a0 con 0x7f507c16ae60 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.082+0000 7f507a7fc700 1 -- 172.21.15.113:0/1401494089 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074012930 con 0x7f507c16ae60 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.083+0000 7f50821b7700 1 -- 172.21.15.113:0/1401494089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c16ae60 msgr2=0x7f507c06efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.083+0000 7f50821b7700 1 --2- 172.21.15.113:0/1401494089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c16ae60 0x7f507c06efc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5074007eb0 tx=0x7f507400d480 comp rx=0 tx=0).stop 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.083+0000 7f50821b7700 1 -- 172.21.15.113:0/1401494089 shutdown_connections 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.083+0000 7f50821b7700 1 --2- 172.21.15.113:0/1401494089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c16ae60 0x7f507c06efc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.083+0000 7f50821b7700 1 --2- 172.21.15.113:0/1401494089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c072bb0 0x7f507c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.083+0000 7f50821b7700 1 -- 172.21.15.113:0/1401494089 >> 172.21.15.113:0/1401494089 conn(0x7f507c0cfa30 msgr2=0x7f507c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.086+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f46ac010c20 con 0x7f46bc0b3d40 2024-01-16T13:52:11.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.086+0000 7f46b97fa700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f46a806b420 0x7f46a806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.086+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46ac08b8a0 con 0x7f46bc0b3d40 2024-01-16T13:52:11.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.087+0000 7f46c0981700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f46a806b420 0x7f46a806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.087+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46bc0a9610 con 0x7f46bc0b3d40 2024-01-16T13:52:11.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.088+0000 7f46c0981700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f46a806b420 0x7f46a806d8e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000c00 tx=0x7f46b0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.088+0000 7f50821b7700 1 -- 172.21.15.113:0/1401494089 shutdown_connections 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.088+0000 7f50821b7700 1 -- 172.21.15.113:0/1401494089 wait complete. 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.090+0000 7f50821b7700 1 Processor -- start 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.091+0000 7f50821b7700 1 -- start start 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f50821b7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f507c0b4090 0x7f507c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f50821b7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c163f60 0x7f507c1611a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f50821b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1663c0 0x7f507c166840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f50821b7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1699a0 con 0x7f507c1663c0 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f50821b7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c166d80 con 0x7f507c0b4090 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f50821b7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c16a4e0 con 0x7f507c163f60 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f507affd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c163f60 0x7f507c1611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f507bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1663c0 0x7f507c166840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f507bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1663c0 0x7f507c166840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40574/0 (socket says 172.21.15.113:40574) 2024-01-16T13:52:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f507bfff700 1 -- 172.21.15.113:0/203592327 learned_addr learned my addr 172.21.15.113:0/203592327 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.092+0000 7f507b7fe700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f507c0b4090 0x7f507c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.094+0000 7f507affd700 1 -- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f507c0b4090 msgr2=0x7f507c0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.094+0000 7f507affd700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f507c0b4090 0x7f507c0aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.094+0000 7f507affd700 1 -- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1663c0 msgr2=0x7f507c166840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.094+0000 7f507affd700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1663c0 0x7f507c166840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.094+0000 7f507affd700 1 -- 172.21.15.113:0/203592327 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5074007b10 con 0x7f507c163f60 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.095+0000 7f507affd700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c163f60 0x7f507c1611a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f507000e870 tx=0x7f507000eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.095+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070013680 con 0x7f507c163f60 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.095+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50700137e0 con 0x7f507c163f60 2024-01-16T13:52:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.095+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070013b70 con 0x7f507c163f60 2024-01-16T13:52:11.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.095+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0af1e0 con 0x7f507c163f60 2024-01-16T13:52:11.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.095+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c169e90 con 0x7f507c163f60 2024-01-16T13:52:11.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.099+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5070013cd0 con 0x7f507c163f60 2024-01-16T13:52:11.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.100+0000 7f5063fff700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f506806b420 0x7f506806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.100+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5070089cd0 con 0x7f507c163f60 2024-01-16T13:52:11.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.096+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46bc0a9610 con 0x7f46bc0b3d40 2024-01-16T13:52:11.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.100+0000 7f507b7fe700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f506806b420 0x7f506806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.101+0000 7f507b7fe700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f506806b420 0x7f506806d8e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5074000f80 tx=0x7f507400b000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.102+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505c0052f0 con 0x7f507c163f60 2024-01-16T13:52:11.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.107+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5070054b20 con 0x7f507c163f60 2024-01-16T13:52:11.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.143+0000 7f5f80c70700 1 Processor -- start 2024-01-16T13:52:11.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.143+0000 7f5f80c70700 1 -- start start 2024-01-16T13:52:11.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7fe1a8479700 1 Processor -- start 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7fe1a8479700 1 -- start start 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7fe1a8479700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe1a01627c0 0x7fe1a00cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7fe1a8479700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a00d0fd0 0x7fe1a0062450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7fe1a8479700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a0062990 con 0x7fe1a0072680 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7fe1a8479700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a006beb0 con 0x7fe1a01627c0 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7fe1a8479700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00c3780 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7f5f80c70700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5f7c0d0520 0x7f5f7c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7f5f80c70700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c071be0 0x7f5f7c0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7f5f80c70700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c15d970 con 0x7f5f7c0b3b50 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7f5f80c70700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c072d60 con 0x7f5f7c0d0520 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.144+0000 7f5f80c70700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c06de80 con 0x7f5f7c071be0 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5f7c0d0520 0x7f5f7c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5f7c0d0520 0x7f5f7c15d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40582/0 (socket says 172.21.15.113:40582) 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 -- 172.21.15.113:0/3878088790 learned_addr learned my addr 172.21.15.113:0/3878088790 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 -- 172.21.15.113:0/3878088790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c071be0 msgr2=0x7f5f7c0b3fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 --2- 172.21.15.113:0/3878088790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c071be0 0x7f5f7c0b3fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 -- 172.21.15.113:0/3878088790 >> v1:172.21.15.113:6789/0 conn(0x7f5f7c0b3b50 legacy=0x7f5f7c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.145+0000 7f5f7a59c700 1 -- 172.21.15.113:0/3878088790 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0723c0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.146+0000 7f5f7a59c700 1 --2- 172.21.15.113:0/3878088790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c15d430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009e80 tx=0x7f5f7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48aa83fd42d5ef44 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.146+0000 7fe1a5a14700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a00d0fd0 0x7fe1a0062450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.146+0000 7fe1a5a14700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a00d0fd0 0x7fe1a0062450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36798/0 (socket says 172.21.15.113:36798) 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.146+0000 7fe1a5a14700 1 -- 172.21.15.113:0/3542092579 learned_addr learned my addr 172.21.15.113:0/3542092579 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.146+0000 7fe1a6215700 1 --2- 172.21.15.113:0/3542092579 >> v2:172.21.15.124:3300/0 conn(0x7fe1a01627c0 0x7fe1a00cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a6215700 1 -- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a00d0fd0 msgr2=0x7fe1a0062450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a6215700 1 --2- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a00d0fd0 0x7fe1a0062450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a6215700 1 -- 172.21.15.113:0/3542092579 >> v1:172.21.15.113:6789/0 conn(0x7fe1a0072680 legacy=0x7fe1a0072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a6215700 1 -- 172.21.15.113:0/3542092579 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00731a0 con 0x7fe1a01627c0 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.146+0000 7f5f7959a700 1 -- 172.21.15.113:0/3878088790 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400e070 con 0x7f5f7c0d0520 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7f5f7959a700 1 -- 172.21.15.113:0/3878088790 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f74003a20 con 0x7f5f7c0d0520 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7f5f7959a700 1 -- 172.21.15.113:0/3878088790 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74003dd0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a6215700 1 --2- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00cf400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe190009840 tx=0x7fe190009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d8b514cb50293c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a5213700 1 -- 172.21.15.113:0/3542092579 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190011830 con 0x7fe1a01627c0 2024-01-16T13:52:11.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a5213700 1 -- 172.21.15.113:0/3542092579 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190011990 con 0x7fe1a01627c0 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.147+0000 7fe1a5213700 1 -- 172.21.15.113:0/3542092579 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190011d00 con 0x7fe1a01627c0 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.148+0000 7f5f80c70700 1 -- 172.21.15.113:0/3878088790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c0d0520 msgr2=0x7f5f7c15d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.148+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3878088790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c15d430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009e80 tx=0x7f5f7400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.148+0000 7f5f80c70700 1 -- 172.21.15.113:0/3878088790 shutdown_connections 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.148+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3878088790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c071be0 0x7f5f7c0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.148+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3878088790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c15d430 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.148+0000 7f5f80c70700 1 -- 172.21.15.113:0/3878088790 >> 172.21.15.113:0/3878088790 conn(0x7f5f7c0cfa30 msgr2=0x7f5f7c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7f5f80c70700 1 -- 172.21.15.113:0/3878088790 shutdown_connections 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7fe1a8479700 1 -- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 msgr2=0x7fe1a00cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7fe1a8479700 1 --2- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00cf400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe190009840 tx=0x7fe190009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7fe1a8479700 1 -- 172.21.15.113:0/3542092579 shutdown_connections 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7fe1a8479700 1 --2- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00cf400 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7fe1a8479700 1 --2- 172.21.15.113:0/3542092579 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a00d0fd0 0x7fe1a0062450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7fe1a8479700 1 -- 172.21.15.113:0/3542092579 >> 172.21.15.113:0/3542092579 conn(0x7fe1a00d02a0 msgr2=0x7fe1a00d06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.149+0000 7f5f80c70700 1 -- 172.21.15.113:0/3878088790 wait complete. 2024-01-16T13:52:11.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.150+0000 7fe1a8479700 1 -- 172.21.15.113:0/3542092579 shutdown_connections 2024-01-16T13:52:11.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.151+0000 7f5f80c70700 1 Processor -- start 2024-01-16T13:52:11.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.150+0000 7fe1a8479700 1 -- 172.21.15.113:0/3542092579 wait complete. 2024-01-16T13:52:11.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.151+0000 7f5f80c70700 1 -- start start 2024-01-16T13:52:11.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f80c70700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c071be0 0x7f5f7c169370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f80c70700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c0b3b50 0x7f5f7c134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7fe1a8479700 1 Processor -- start 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f80c70700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c0cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f80c70700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c16a940 con 0x7f5f7c071be0 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f80c70700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c169cc0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f80c70700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c169fa0 con 0x7f5f7c0b3b50 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f79d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c0b3b50 0x7f5f7c134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f79d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c0b3b50 0x7f5f7c134ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36804/0 (socket says 172.21.15.113:36804) 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f7ad9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c0cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f79d9b700 1 -- 172.21.15.113:0/3487629363 learned_addr learned my addr 172.21.15.113:0/3487629363 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.152+0000 7f5f7a59c700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c071be0 0x7f5f7c169370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.153+0000 7fe1a8479700 1 -- start start 2024-01-16T13:52:11.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.153+0000 7f5f7ad9d700 1 -- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c0b3b50 msgr2=0x7f5f7c134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.153+0000 7f5f7ad9d700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c0b3b50 0x7f5f7c134ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.153+0000 7f5f7ad9d700 1 -- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c071be0 msgr2=0x7f5f7c169370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.153+0000 7f5f7ad9d700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c071be0 0x7f5f7c169370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.153+0000 7f5f7ad9d700 1 -- 172.21.15.113:0/3487629363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f74009b30 con 0x7f5f7c0d0520 2024-01-16T13:52:11.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7f5f7ad9d700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c0cdcb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000b390 tx=0x7f5f7000b6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7fe1a8479700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a0072680 0x7fe1a015f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7fe1a8479700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 0x7fe1a0154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7fe1a8479700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7fe1a8479700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a015fd40 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7fe1a8479700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a01640c0 con 0x7fe1a01627c0 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.154+0000 7fe1a8479700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00b2de0 con 0x7fe1a0072680 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 0x7fe1a0154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 0x7fe1a0154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40608/0 (socket says 172.21.15.113:40608) 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 -- 172.21.15.113:0/2433346408 learned_addr learned my addr 172.21.15.113:0/2433346408 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a6215700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a0072680 0x7fe1a015f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a6a16700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 -- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a0072680 msgr2=0x7fe1a015f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a0072680 0x7fe1a015f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 -- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 msgr2=0x7fe1a00af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00af900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 -- 172.21.15.113:0/2433346408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900094a0 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.155+0000 7fe1a5a14700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 0x7fe1a0154ee0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00e7e0 tx=0x7fe19c00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c013610 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c013770 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c013b00 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70011620 con 0x7f5f7c0d0520 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f70011780 con 0x7f5f7c0d0520 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70011b30 con 0x7f5f7c0d0520 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c16a280 con 0x7f5f7c0d0520 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c1698b0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.156+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a00b5720 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.157+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a00b3080 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.158+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a00b32a0 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.159+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f600052f0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.159+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe19c013c60 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.160+0000 7fe196ffd700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe18406b420 0x7fe18406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.160+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe19c089770 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.160+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5f70011c90 con 0x7f5f7c0d0520 2024-01-16T13:52:11.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.160+0000 7f5f6affd700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f5c06b420 0x7f5f5c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.161+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f70089590 con 0x7f5f7c0d0520 2024-01-16T13:52:11.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.162+0000 7fe1a6215700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe18406b420 0x7fe18406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.162+0000 7f5f7a59c700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f5c06b420 0x7f5f5c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.162+0000 7fc9c0fc0700 1 Processor -- start 2024-01-16T13:52:11.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.162+0000 7fe1a6215700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe18406b420 0x7fe18406d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00b0dd0 tx=0x7fe190008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.162+0000 7f5f7a59c700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f5c06b420 0x7f5f5c06d8e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f74000f80 tx=0x7f5f74009000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.162+0000 7fc9c0fc0700 1 -- start start 2024-01-16T13:52:11.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.163+0000 7fc9c0fc0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 0x7fc9bc0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.163+0000 7fc9c0fc0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc9bc071be0 0x7fc9bc140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.164+0000 7fc9c0fc0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9bc0720e0 con 0x7fc9bc0b4090 2024-01-16T13:52:11.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.164+0000 7fc9c0fc0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9bc072d60 con 0x7fc9bc071be0 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.164+0000 7fc9c0fc0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc06de80 con 0x7fc9bc15d6a0 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.164+0000 7fc9ba59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 0x7fc9bc0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.164+0000 7fc9ba59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 0x7fc9bc0b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36826/0 (socket says 172.21.15.113:36826) 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.164+0000 7fc9ba59c700 1 -- 172.21.15.113:0/4155754441 learned_addr learned my addr 172.21.15.113:0/4155754441 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.165+0000 7fc9bad9d700 1 --2- 172.21.15.113:0/4155754441 >> v2:172.21.15.124:3300/0 conn(0x7fc9bc071be0 0x7fc9bc140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.165+0000 7fc9ba59c700 1 -- 172.21.15.113:0/4155754441 >> v2:172.21.15.124:3300/0 conn(0x7fc9bc071be0 msgr2=0x7fc9bc140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.165+0000 7fc9ba59c700 1 --2- 172.21.15.113:0/4155754441 >> v2:172.21.15.124:3300/0 conn(0x7fc9bc071be0 0x7fc9bc140fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.165+0000 7fc9ba59c700 1 -- 172.21.15.113:0/4155754441 >> v1:172.21.15.113:6789/0 conn(0x7fc9bc0b4090 legacy=0x7fc9bc0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.165+0000 7fc9ba59c700 1 -- 172.21.15.113:0/4155754441 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc0723c0 con 0x7fc9bc15d6a0 2024-01-16T13:52:11.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.165+0000 7fc9ba59c700 1 --2- 172.21.15.113:0/4155754441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 0x7fc9bc0b3b50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009d40 tx=0x7fc9b400c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4ed7dab13f83f54 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.166+0000 7fc9b959a700 1 -- 172.21.15.113:0/4155754441 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b400cc90 con 0x7fc9bc15d6a0 2024-01-16T13:52:11.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.166+0000 7fc9b959a700 1 -- 172.21.15.113:0/4155754441 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b400cdf0 con 0x7fc9bc15d6a0 2024-01-16T13:52:11.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.166+0000 7fc9b959a700 1 -- 172.21.15.113:0/4155754441 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4003c20 con 0x7fc9bc15d6a0 2024-01-16T13:52:11.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.166+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe19c0545c0 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/4155754441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 msgr2=0x7fc9bc0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/4155754441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 0x7fc9bc0b3b50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009d40 tx=0x7fc9b400c440 comp rx=0 tx=0).stop 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/4155754441 shutdown_connections 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/4155754441 >> v2:172.21.15.124:3300/0 conn(0x7fc9bc071be0 0x7fc9bc140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/4155754441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc15d6a0 0x7fc9bc0b3b50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/4155754441 >> 172.21.15.113:0/4155754441 conn(0x7fc9bc0cfa30 msgr2=0x7fc9bc0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/4155754441 shutdown_connections 2024-01-16T13:52:11.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.167+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/4155754441 wait complete. 2024-01-16T13:52:11.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.169+0000 7fc9c0fc0700 1 Processor -- start 2024-01-16T13:52:11.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.170+0000 7fc9c0fc0700 1 -- start start 2024-01-16T13:52:11.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9c0fc0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 0x7fc9bc12f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9c0fc0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 0x7fc9bc1ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9c0fc0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9bc15d6a0 0x7fc9bc0afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9c0fc0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc154ef0 con 0x7fc9bc15d6a0 2024-01-16T13:52:11.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9c0fc0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc12ff30 con 0x7fc9bc071be0 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9c0fc0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc159050 con 0x7fc9bc0b4090 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9b9d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 0x7fc9bc1ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9ba59c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 0x7fc9bc12f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9b9d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 0x7fc9bc1ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36830/0 (socket says 172.21.15.113:36830) 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9b9d9b700 1 -- 172.21.15.113:0/2765918867 learned_addr learned my addr 172.21.15.113:0/2765918867 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.171+0000 7fc9ba59c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 0x7fc9bc12f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57810/0 (socket says 172.21.15.113:57810) 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9bad9d700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9bc15d6a0 0x7fc9bc0afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9b9d9b700 1 -- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 msgr2=0x7fc9bc12f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9b9d9b700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 0x7fc9bc12f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9b9d9b700 1 -- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9bc15d6a0 msgr2=0x7fc9bc0afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9b9d9b700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9bc15d6a0 0x7fc9bc0afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9b9d9b700 1 -- 172.21.15.113:0/2765918867 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b400d040 con 0x7fc9bc0b4090 2024-01-16T13:52:11.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9ba59c700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 0x7fc9bc12f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:11.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.172+0000 7fc9b9d9b700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 0x7fc9bc1ffcc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00c4a0 tx=0x7fc9ac00c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.166+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f70054470 con 0x7f5f7c0d0520 2024-01-16T13:52:11.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.181+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0054e0 con 0x7fc9bc0b4090 2024-01-16T13:52:11.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.181+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac003680 con 0x7fc9bc0b4090 2024-01-16T13:52:11.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.181+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac003a30 con 0x7fc9bc0b4090 2024-01-16T13:52:11.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.181+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc1548a0 con 0x7fc9bc0b4090 2024-01-16T13:52:11.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.181+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc140870 con 0x7fc9bc0b4090 2024-01-16T13:52:11.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.183+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc9ac003b90 con 0x7fc9bc0b4090 2024-01-16T13:52:11.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.184+0000 7fc9aaffd700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc9a406b5c0 0x7fc9a406da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.184+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9ac088f90 con 0x7fc9bc0b4090 2024-01-16T13:52:11.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.184+0000 7fc9ba59c700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc9a406b5c0 0x7fc9a406da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.185+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a00052f0 con 0x7fc9bc0b4090 2024-01-16T13:52:11.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.186+0000 7fc9ba59c700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc9a406b5c0 0x7fc9a406da80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4000f80 tx=0x7fc9b4012000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.186+0000 7f235bc24700 1 Processor -- start 2024-01-16T13:52:11.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.186+0000 7f235bc24700 1 -- start start 2024-01-16T13:52:11.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.187+0000 7f235bc24700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23541456b0 0x7f23540b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.187+0000 7f235bc24700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235406f4e0 con 0x7f2354072850 2024-01-16T13:52:11.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.187+0000 7f235bc24700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23540cc8c0 con 0x7f23540cc3c0 2024-01-16T13:52:11.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.187+0000 7f235bc24700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2354145a90 con 0x7f23541456b0 2024-01-16T13:52:11.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23541456b0 0x7f23540b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23541456b0 0x7f23540b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57818/0 (socket says 172.21.15.113:57818) 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 -- 172.21.15.113:0/1496170481 learned_addr learned my addr 172.21.15.113:0/1496170481 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 -- 172.21.15.113:0/1496170481 >> v1:172.21.15.124:6789/0 conn(0x7f23540cc3c0 legacy=0x7f23540b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 -- 172.21.15.113:0/1496170481 >> v1:172.21.15.113:6789/0 conn(0x7f2354072850 legacy=0x7f23540cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 -- 172.21.15.113:0/1496170481 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235406ef20 con 0x7f23541456b0 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.188+0000 7f23599c0700 1 --2- 172.21.15.113:0/1496170481 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23541456b0 0x7f23540b3b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f234c009e50 tx=0x7f234c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=820826f3424bec2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.189+0000 7f23589be700 1 -- 172.21.15.113:0/1496170481 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c00e070 con 0x7f23541456b0 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.189+0000 7f23589be700 1 -- 172.21.15.113:0/1496170481 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c003a20 con 0x7f23541456b0 2024-01-16T13:52:11.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.189+0000 7f23589be700 1 -- 172.21.15.113:0/1496170481 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c003db0 con 0x7f23541456b0 2024-01-16T13:52:11.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.190+0000 7f235bc24700 1 -- 172.21.15.113:0/1496170481 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23541456b0 msgr2=0x7f23540b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.190+0000 7f235bc24700 1 --2- 172.21.15.113:0/1496170481 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23541456b0 0x7f23540b3b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f234c009e50 tx=0x7f234c00b330 comp rx=0 tx=0).stop 2024-01-16T13:52:11.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.190+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9ac053e90 con 0x7fc9bc0b4090 2024-01-16T13:52:11.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.190+0000 7f235bc24700 1 -- 172.21.15.113:0/1496170481 shutdown_connections 2024-01-16T13:52:11.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.190+0000 7f235bc24700 1 --2- 172.21.15.113:0/1496170481 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23541456b0 0x7f23540b3b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.190+0000 7f235bc24700 1 -- 172.21.15.113:0/1496170481 >> 172.21.15.113:0/1496170481 conn(0x7f23540cfa30 msgr2=0x7f23540cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.194+0000 7f235bc24700 1 -- 172.21.15.113:0/1496170481 shutdown_connections 2024-01-16T13:52:11.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.194+0000 7f235bc24700 1 -- 172.21.15.113:0/1496170481 wait complete. 2024-01-16T13:52:11.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.196+0000 7f235bc24700 1 Processor -- start 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.197+0000 7f235bc24700 1 -- start start 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f235bc24700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2354072850 0x7f235406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f235bc24700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 0x7f235406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f235bc24700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23541456b0 0x7f23541ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f235bc24700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354133a10 con 0x7f23540cc3c0 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f235bc24700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354133700 con 0x7f2354072850 2024-01-16T13:52:11.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f235bc24700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23541419b0 con 0x7f23541456b0 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f23591bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 0x7f235406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f23591bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 0x7f235406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40638/0 (socket says 172.21.15.113:40638) 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.198+0000 7f23591bf700 1 -- 172.21.15.113:0/4294037034 learned_addr learned my addr 172.21.15.113:0/4294037034 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.199+0000 7f23591bf700 1 -- 172.21.15.113:0/4294037034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23541456b0 msgr2=0x7f23541ffbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.199+0000 7f23591bf700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23541456b0 0x7f23541ffbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.199+0000 7f23591bf700 1 -- 172.21.15.113:0/4294037034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2354072850 msgr2=0x7f235406d7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.199+0000 7f23591bf700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2354072850 0x7f235406d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.199+0000 7f23591bf700 1 -- 172.21.15.113:0/4294037034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c009b30 con 0x7f23540cc3c0 2024-01-16T13:52:11.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.199+0000 7f23591bf700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 0x7f235406dd10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f234400c4e0 tx=0x7f234400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.203+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440052d0 con 0x7f23540cc3c0 2024-01-16T13:52:11.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.203+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2344005430 con 0x7f23540cc3c0 2024-01-16T13:52:11.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.203+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440038c0 con 0x7f23540cc3c0 2024-01-16T13:52:11.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.203+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23541318e0 con 0x7f23540cc3c0 2024-01-16T13:52:11.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.203+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2354131ea0 con 0x7f23540cc3c0 2024-01-16T13:52:11.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.209+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c0052f0 con 0x7f23540cc3c0 2024-01-16T13:52:11.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.219+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2344003a20 con 0x7f23540cc3c0 2024-01-16T13:52:11.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.220+0000 7f234a7fc700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f233806b420 0x7f233806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:11.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.220+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2344089080 con 0x7f23540cc3c0 2024-01-16T13:52:11.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.220+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f234408b5d0 con 0x7f23540cc3c0 2024-01-16T13:52:11.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.221+0000 7f23599c0700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f233806b420 0x7f233806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:11.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.221+0000 7f23599c0700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f233806b420 0x7f233806d8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f234c000f80 tx=0x7f234c009000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:11.457 INFO:teuthology.orchestra.run.smithi113.stdout:55834574851 2024-01-16T13:52:11.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.472+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f46bc0a9610 con 0x7f46bc0b3d40 2024-01-16T13:52:11.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.473+0000 7f46b97fa700 1 -- 172.21.15.113:0/4080227822 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f46bc0a9610 con 0x7f46bc0b3d40 2024-01-16T13:52:11.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.476+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f46a806b420 msgr2=0x7f46a806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.476+0000 7f46c2be5700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f46a806b420 0x7f46a806d8e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000c00 tx=0x7f46b0009040 comp rx=0 tx=0).stop 2024-01-16T13:52:11.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.476+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 msgr2=0x7f46bc15d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.476+0000 7f46c2be5700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc15d430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0737d0 tx=0x7f46ac004750 comp rx=0 tx=0).stop 2024-01-16T13:52:11.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.477+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 shutdown_connections 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.477+0000 7f46c2be5700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f46a806b420 0x7f46a806d8e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.477+0000 7f46c2be5700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46bc140680 0x7f46bc15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.477+0000 7f46c2be5700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46bc070d60 0x7f46bc0ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.477+0000 7f46c2be5700 1 --2- 172.21.15.113:0/4080227822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46bc0b3d40 0x7f46bc15d430 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.477+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 >> 172.21.15.113:0/4080227822 conn(0x7f46bc0b1740 msgr2=0x7f46bc1420d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.478+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 shutdown_connections 2024-01-16T13:52:11.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.478+0000 7f46c2be5700 1 -- 172.21.15.113:0/4080227822 wait complete. 2024-01-16T13:52:11.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.487+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9c7c0055a0 con 0x7f9c9c071a10 2024-01-16T13:52:11.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.487+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f70c4142190 con 0x7f70c40ce140 2024-01-16T13:52:11.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.487+0000 7f70b9ffb700 1 -- 172.21.15.113:0/3772403196 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f70c4142190 con 0x7f70c40ce140 2024-01-16T13:52:11.469 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:11.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.488+0000 7f9c91ffb700 1 -- 172.21.15.113:0/2296191464 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9c980597c0 con 0x7f9c9c071a10 2024-01-16T13:52:11.472 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:11.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.491+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f70a806b420 msgr2=0x7f70a806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.491+0000 7f70cb627700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f70a806b420 0x7f70a806d8e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70c40b1360 tx=0x7f70b4008040 comp rx=0 tx=0).stop 2024-01-16T13:52:11.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.491+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 msgr2=0x7f70c4141a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.491+0000 7f70cb627700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 0x7f70c4141a70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f70c4073510 tx=0x7f70bc004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 shutdown_connections 2024-01-16T13:52:11.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f70a806b420 0x7f70a806d8e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70c415f460 0x7f70c41fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70c40ce140 0x7f70c4141a70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 --2- 172.21.15.113:0/3772403196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70c415af10 0x7f70c41fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 >> 172.21.15.113:0/3772403196 conn(0x7f70c40cd570 msgr2=0x7f70c4073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.492+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 shutdown_connections 2024-01-16T13:52:11.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.493+0000 7f70cb627700 1 -- 172.21.15.113:0/3772403196 wait complete. 2024-01-16T13:52:11.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.493+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c8006b420 msgr2=0x7f9c8006d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.493+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c8006b420 0x7f9c8006d8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0085a0 tx=0x7f9c8c0084f0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.493+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 msgr2=0x7f9c9c1393f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.493+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 0x7f9c9c1393f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9c98014040 tx=0x7f9c98004750 comp rx=0 tx=0).stop 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 shutdown_connections 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c8006b420 0x7f9c8006d8e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c9c15d6a0 0x7f9c9c158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c9c071a10 0x7f9c9c1393f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 --2- 172.21.15.113:0/2296191464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c9c0b4090 0x7f9c9c156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 >> 172.21.15.113:0/2296191464 conn(0x7f9c9c0cfa30 msgr2=0x7f9c9c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 shutdown_connections 2024-01-16T13:52:11.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.494+0000 7f9ca378f700 1 -- 172.21.15.113:0/2296191464 wait complete. 2024-01-16T13:52:11.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2024-01-16T13:52:11.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.507+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f505c0055a0 con 0x7f507c163f60 2024-01-16T13:52:11.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.508+0000 7f5063fff700 1 -- 172.21.15.113:0/203592327 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5070021030 con 0x7f507c163f60 2024-01-16T13:52:11.493 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f506806b420 msgr2=0x7f506806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f506806b420 0x7f506806d8e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5074000f80 tx=0x7f507400b000 comp rx=0 tx=0).stop 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c163f60 msgr2=0x7f507c1611a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c163f60 0x7f507c1611a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f507000e870 tx=0x7f507000eb80 comp rx=0 tx=0).stop 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 shutdown_connections 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f506806b420 0x7f506806d8e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f507c1663c0 0x7f507c166840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f507c0b4090 0x7f507c0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 --2- 172.21.15.113:0/203592327 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f507c163f60 0x7f507c1611a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.512+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 >> 172.21.15.113:0/203592327 conn(0x7f507c0cfa30 msgr2=0x7f507c167220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.513+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 shutdown_connections 2024-01-16T13:52:11.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.513+0000 7f50821b7700 1 -- 172.21.15.113:0/203592327 wait complete. 2024-01-16T13:52:11.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-01-16T13:52:11.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-01-16T13:52:11.508 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-01-16T13:52:11.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.539+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe1a0067f60 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.540+0000 7fe196ffd700 1 -- 172.21.15.113:0/2433346408 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe19c00c470 con 0x7fe1a00d0fd0 2024-01-16T13:52:11.522 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:11.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe18406b420 msgr2=0x7fe18406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe18406b420 0x7fe18406d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00b0dd0 tx=0x7fe190008030 comp rx=0 tx=0).stop 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 msgr2=0x7fe1a0154ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 0x7fe1a0154ee0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00e7e0 tx=0x7fe19c00eba0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 shutdown_connections 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe18406b420 0x7fe18406d8e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1a01627c0 0x7fe1a00af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1a0072680 0x7fe1a015f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 --2- 172.21.15.113:0/2433346408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1a00d0fd0 0x7fe1a0154ee0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 >> 172.21.15.113:0/2433346408 conn(0x7fe1a00d02a0 msgr2=0x7fe1a00c2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 shutdown_connections 2024-01-16T13:52:11.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.542+0000 7fe1a8479700 1 -- 172.21.15.113:0/2433346408 wait complete. 2024-01-16T13:52:11.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2024-01-16T13:52:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.566+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f233c0055a0 con 0x7f23540cc3c0 2024-01-16T13:52:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.566+0000 7f234a7fc700 1 -- 172.21.15.113:0/4294037034 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2344053f60 con 0x7f23540cc3c0 2024-01-16T13:52:11.548 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.567+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5f600055a0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.568+0000 7f5f6affd700 1 -- 172.21.15.113:0/3487629363 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f700091d0 con 0x7f5f7c0d0520 2024-01-16T13:52:11.550 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f233806b420 msgr2=0x7f233806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f233806b420 0x7f233806d8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f234c000f80 tx=0x7f234c009000 comp rx=0 tx=0).stop 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 msgr2=0x7f235406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 0x7f235406dd10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f234400c4e0 tx=0x7f234400c8a0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 shutdown_connections 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f233806b420 0x7f233806d8e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23541456b0 0x7f23541ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2354072850 0x7f235406d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 --2- 172.21.15.113:0/4294037034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23540cc3c0 0x7f235406dd10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 >> 172.21.15.113:0/4294037034 conn(0x7f23540cfa30 msgr2=0x7f23541576c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 shutdown_connections 2024-01-16T13:52:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.569+0000 7f235bc24700 1 -- 172.21.15.113:0/4294037034 wait complete. 2024-01-16T13:52:11.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f5c06b420 msgr2=0x7f5f5c06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f5c06b420 0x7f5f5c06d8e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f74000f80 tx=0x7f5f74009000 comp rx=0 tx=0).stop 2024-01-16T13:52:11.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f7c0d0520 msgr2=0x7f5f7c0cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c0cdcb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000b390 tx=0x7f5f7000b6a0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 shutdown_connections 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5f5c06b420 0x7f5f5c06d8e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f7c0d0520 0x7f5f7c0cdcb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f7c071be0 0x7f5f7c169370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 --2- 172.21.15.113:0/3487629363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f7c0b3b50 0x7f5f7c134ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 >> 172.21.15.113:0/3487629363 conn(0x7f5f7c0cfa30 msgr2=0x7f5f7c0b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 shutdown_connections 2024-01-16T13:52:11.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.571+0000 7f5f80c70700 1 -- 172.21.15.113:0/3487629363 wait complete. 2024-01-16T13:52:11.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2024-01-16T13:52:11.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-01-16T13:52:11.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.591+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc9a00055a0 con 0x7fc9bc0b4090 2024-01-16T13:52:11.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.592+0000 7fc9aaffd700 1 -- 172.21.15.113:0/2765918867 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc9ac017070 con 0x7fc9bc0b4090 2024-01-16T13:52:11.574 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc9a406b5c0 msgr2=0x7fc9a406da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc9a406b5c0 0x7fc9a406da80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4000f80 tx=0x7fc9b4012000 comp rx=0 tx=0).stop 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 msgr2=0x7fc9bc1ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 0x7fc9bc1ffcc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00c4a0 tx=0x7fc9ac00c860 comp rx=0 tx=0).stop 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 shutdown_connections 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc9a406b5c0 0x7fc9a406da80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9bc15d6a0 0x7fc9bc0afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9bc071be0 0x7fc9bc12f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 --2- 172.21.15.113:0/2765918867 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9bc0b4090 0x7fc9bc1ffcc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 >> 172.21.15.113:0/2765918867 conn(0x7fc9bc0cfa30 msgr2=0x7fc9bc0cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:11.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.594+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 shutdown_connections 2024-01-16T13:52:11.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:11.595+0000 7fc9c0fc0700 1 -- 172.21.15.113:0/2765918867 wait complete. 2024-01-16T13:52:11.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-01-16T13:52:12.480 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T13:52:12.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T13:52:12.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T13:52:12.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T13:52:12.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T13:52:12.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T13:52:12.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T13:52:12.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T13:52:12.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.679+0000 7fd15f7a7700 1 Processor -- start 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.679+0000 7fd15f7a7700 1 -- start start 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.680+0000 7fd15f7a7700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd1580d1570 0x7fd1580d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.680+0000 7fd15f7a7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd1580b2fc0 0x7fd1580d3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.680+0000 7fd15f7a7700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1580d2d10 con 0x7fd15815dfc0 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.680+0000 7fd15f7a7700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1580cefd0 con 0x7fd1580d1570 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.680+0000 7fd15f7a7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580cf2b0 con 0x7fd1580b2fc0 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15d543700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd15815dfc0 0x7fd1580d0e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:59162/0 (socket says 172.21.15.113:59162) 2024-01-16T13:52:12.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15d543700 1 -- 172.21.15.113:0/2993590128 learned_addr learned my addr 172.21.15.113:0/2993590128 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:12.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15cd42700 1 --2- 172.21.15.113:0/2993590128 >> v2:172.21.15.124:3300/0 conn(0x7fd1580d1570 0x7fd1580d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15cd42700 1 -- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd1580b2fc0 msgr2=0x7fd1580d3a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15cd42700 1 --2- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd1580b2fc0 0x7fd1580d3a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15cd42700 1 -- 172.21.15.113:0/2993590128 >> v1:172.21.15.124:6789/0 conn(0x7fd15815dfc0 legacy=0x7fd1580d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.681+0000 7fd15cd42700 1 -- 172.21.15.113:0/2993590128 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580d2fb0 con 0x7fd1580d1570 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.682+0000 7fd15cd42700 1 --2- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1580d1570 0x7fd1580d34d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd148009840 tx=0x7fd148009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d5e15d0e260178 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.682+0000 7fd14ffff700 1 -- 172.21.15.113:0/2993590128 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148011830 con 0x7fd1580d1570 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.682+0000 7fd14ffff700 1 -- 172.21.15.113:0/2993590128 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd148011990 con 0x7fd1580d1570 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.682+0000 7fd14ffff700 1 -- 172.21.15.113:0/2993590128 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148011d20 con 0x7fd1580d1570 2024-01-16T13:52:12.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 -- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1580d1570 msgr2=0x7fd1580d34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1580d1570 0x7fd1580d34d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd148009840 tx=0x7fd148009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:12.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 -- 172.21.15.113:0/2993590128 shutdown_connections 2024-01-16T13:52:12.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd1580b2fc0 0x7fd1580d3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/2993590128 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1580d1570 0x7fd1580d34d0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd148009840 tx=0x7fd148009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:12.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 -- 172.21.15.113:0/2993590128 >> 172.21.15.113:0/2993590128 conn(0x7fd15806b410 msgr2=0x7fd15806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 -- 172.21.15.113:0/2993590128 shutdown_connections 2024-01-16T13:52:12.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.683+0000 7fd15f7a7700 1 -- 172.21.15.113:0/2993590128 wait complete. 2024-01-16T13:52:12.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.685+0000 7fd15f7a7700 1 Processor -- start 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.685+0000 7fd15f7a7700 1 -- start start 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.686+0000 7fd15f7a7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1580b2fc0 0x7fd1581f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.686+0000 7fd15f7a7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd15815dfc0 0x7fd158153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.686+0000 7fd15f7a7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1581f87c0 0x7fd1580d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.686+0000 7fd15f7a7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158151030 con 0x7fd1580b2fc0 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.686+0000 7fd15f7a7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1581f2ce0 con 0x7fd1581f87c0 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.686+0000 7fd15f7a7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd158154010 con 0x7fd15815dfc0 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15dd44700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1581f87c0 0x7fd1580d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15dd44700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1581f87c0 0x7fd1580d34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57844/0 (socket says 172.21.15.113:57844) 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15dd44700 1 -- 172.21.15.113:0/4124231166 learned_addr learned my addr 172.21.15.113:0/4124231166 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15d543700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1580b2fc0 0x7fd1581f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15d543700 1 -- 172.21.15.113:0/4124231166 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd15815dfc0 msgr2=0x7fd158153ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:12.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15d543700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd15815dfc0 0x7fd158153ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15d543700 1 -- 172.21.15.113:0/4124231166 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1581f87c0 msgr2=0x7fd1580d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15d543700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1581f87c0 0x7fd1580d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.687+0000 7fd15d543700 1 -- 172.21.15.113:0/4124231166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480094a0 con 0x7fd1580b2fc0 2024-01-16T13:52:12.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.688+0000 7fd15d543700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1580b2fc0 0x7fd1581f27a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c500 tx=0x7fd15400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.688+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154005d70 con 0x7fd1580b2fc0 2024-01-16T13:52:12.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.688+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1540070d0 con 0x7fd1580b2fc0 2024-01-16T13:52:12.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.688+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154007440 con 0x7fd1580b2fc0 2024-01-16T13:52:12.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.688+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1581f8d40 con 0x7fd1580b2fc0 2024-01-16T13:52:12.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.688+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1581f9300 con 0x7fd1580b2fc0 2024-01-16T13:52:12.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.690+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd158067f60 con 0x7fd1580b2fc0 2024-01-16T13:52:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.701+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd1540075a0 con 0x7fd1580b2fc0 2024-01-16T13:52:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.702+0000 7fd14dffb700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd13c06b420 0x7fd13c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.702+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd154088e40 con 0x7fd1580b2fc0 2024-01-16T13:52:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.702+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd15408d450 con 0x7fd1580b2fc0 2024-01-16T13:52:12.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.708+0000 7f353a20f700 1 Processor -- start 2024-01-16T13:52:12.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.708+0000 7f353a20f700 1 -- start start 2024-01-16T13:52:12.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f353a20f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f353415d6a0 0x7f35340b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f353a20f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3534071be0 0x7f3534140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f353a20f700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35340720e0 con 0x7f35340b4090 2024-01-16T13:52:12.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7fd15cd42700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd13c06b420 0x7fd13c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f353a20f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3534072d60 con 0x7f353415d6a0 2024-01-16T13:52:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f353a20f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f353406de80 con 0x7f3534071be0 2024-01-16T13:52:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f353415d6a0 0x7f35340b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f353415d6a0 0x7f35340b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40662/0 (socket says 172.21.15.113:40662) 2024-01-16T13:52:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 -- 172.21.15.113:0/762597395 learned_addr learned my addr 172.21.15.113:0/762597395 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 -- 172.21.15.113:0/762597395 >> v2:172.21.15.124:3300/0 conn(0x7f3534071be0 msgr2=0x7f3534140fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T13:52:12.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 --2- 172.21.15.113:0/762597395 >> v2:172.21.15.124:3300/0 conn(0x7f3534071be0 0x7f3534140fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 -- 172.21.15.113:0/762597395 >> v1:172.21.15.124:6789/0 conn(0x7f35340b4090 legacy=0x7f35340714d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T13:52:12.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 -- 172.21.15.113:0/762597395 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340723c0 con 0x7f353415d6a0 2024-01-16T13:52:12.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7fd15cd42700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd13c06b420 0x7fd13c06d8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd148008000 tx=0x7fd148008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.710+0000 7f35337fe700 1 --2- 172.21.15.113:0/762597395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f353415d6a0 0x7f35340b3b50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3534070a50 tx=0x7f352400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c4c4ba718e0d6e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.711+0000 7f35327fc700 1 -- 172.21.15.113:0/762597395 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524010070 con 0x7f353415d6a0 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.711+0000 7f35327fc700 1 -- 172.21.15.113:0/762597395 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352400da40 con 0x7f353415d6a0 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.711+0000 7f35327fc700 1 -- 172.21.15.113:0/762597395 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352400ddd0 con 0x7f353415d6a0 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.712+0000 7f353a20f700 1 -- 172.21.15.113:0/762597395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f353415d6a0 msgr2=0x7f35340b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.712+0000 7f353a20f700 1 --2- 172.21.15.113:0/762597395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f353415d6a0 0x7f35340b3b50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3534070a50 tx=0x7f352400d330 comp rx=0 tx=0).stop 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.713+0000 7f353a20f700 1 -- 172.21.15.113:0/762597395 shutdown_connections 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.713+0000 7f353a20f700 1 --2- 172.21.15.113:0/762597395 >> v2:172.21.15.124:3300/0 conn(0x7f3534071be0 0x7f3534140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.713+0000 7f353a20f700 1 --2- 172.21.15.113:0/762597395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f353415d6a0 0x7f35340b3b50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.713+0000 7f353a20f700 1 -- 172.21.15.113:0/762597395 >> 172.21.15.113:0/762597395 conn(0x7f35340cfa30 msgr2=0x7f35340cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.713+0000 7f353a20f700 1 -- 172.21.15.113:0/762597395 shutdown_connections 2024-01-16T13:52:12.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.713+0000 7f353a20f700 1 -- 172.21.15.113:0/762597395 wait complete. 2024-01-16T13:52:12.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.714+0000 7f353a20f700 1 Processor -- start 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.714+0000 7f353a20f700 1 -- start start 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f353a20f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3534071be0 0x7f35340aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f353a20f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 0x7f35340cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f353a20f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35340cdc10 0x7f35341f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f353a20f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35340cc960 con 0x7f35340cdc10 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f353a20f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35340cd780 con 0x7f3534071be0 2024-01-16T13:52:12.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f353a20f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35340ccb70 con 0x7f353415d6a0 2024-01-16T13:52:12.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f3532ffd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 0x7f35340cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f3532ffd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 0x7f35340cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36848/0 (socket says 172.21.15.113:36848) 2024-01-16T13:52:12.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f3532ffd700 1 -- 172.21.15.113:0/3909356736 learned_addr learned my addr 172.21.15.113:0/3909356736 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.715+0000 7f3533fff700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35340cdc10 0x7f35341f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3532ffd700 1 -- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3534071be0 msgr2=0x7f35340aa2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3532ffd700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3534071be0 0x7f35340aa2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3532ffd700 1 -- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35340cdc10 msgr2=0x7f35341f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3532ffd700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35340cdc10 0x7f35341f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3532ffd700 1 -- 172.21.15.113:0/3909356736 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35240085d0 con 0x7f353415d6a0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3533fff700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35340cdc10 0x7f35341f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f3532ffd700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 0x7f35340cc420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f352800b470 tx=0x7f352800b830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528011830 con 0x7f353415d6a0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35340cce50 con 0x7f353415d6a0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528011990 con 0x7f353415d6a0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528011d40 con 0x7f353415d6a0 2024-01-16T13:52:12.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.716+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35340cd410 con 0x7f353415d6a0 2024-01-16T13:52:12.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.720+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35340751e0 con 0x7f353415d6a0 2024-01-16T13:52:12.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.721+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3528004020 con 0x7f353415d6a0 2024-01-16T13:52:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.721+0000 7f351bfff700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f352006b380 0x7f352006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.721+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35280898f0 con 0x7f353415d6a0 2024-01-16T13:52:12.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.722+0000 7f35337fe700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f352006b380 0x7f352006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.722+0000 7f35337fe700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f352006b380 0x7f352006d840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f352400d370 tx=0x7f3524012000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.726+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3528054760 con 0x7f353415d6a0 2024-01-16T13:52:12.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.730+0000 7f27e5f39700 1 Processor -- start 2024-01-16T13:52:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.731+0000 7f27e5f39700 1 -- start start 2024-01-16T13:52:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.732+0000 7f27e5f39700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e0163480 0x7f27e00b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.732+0000 7f27e5f39700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27e00b2550 0x7f27e00b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.732+0000 7f27e5f39700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e00b2e80 con 0x7f27e0163860 2024-01-16T13:52:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.732+0000 7f27e5f39700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e00b29c0 con 0x7f27e00b2550 2024-01-16T13:52:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.732+0000 7f27e5f39700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e0164510 con 0x7f27e0163480 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27e00b2550 0x7f27e00b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f27e00b2550 0x7f27e00b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40690/0 (socket says 172.21.15.113:40690) 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 -- 172.21.15.113:0/2690093491 learned_addr learned my addr 172.21.15.113:0/2690093491 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 -- 172.21.15.113:0/2690093491 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e0163480 msgr2=0x7f27e00b01a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 --2- 172.21.15.113:0/2690093491 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e0163480 0x7f27e00b01a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 -- 172.21.15.113:0/2690093491 >> v1:172.21.15.113:6789/0 conn(0x7f27e0163860 legacy=0x7f27e00b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.737+0000 7f27dffff700 1 -- 172.21.15.113:0/2690093491 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e0160610 con 0x7f27e00b2550 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.738+0000 7f27dffff700 1 --2- 172.21.15.113:0/2690093491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00b2550 0x7f27e00b06e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009e50 tx=0x7f27d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4f49bf7fabb4fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.738+0000 7f27de7fc700 1 -- 172.21.15.113:0/2690093491 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d800e070 con 0x7f27e00b2550 2024-01-16T13:52:12.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.738+0000 7f27de7fc700 1 -- 172.21.15.113:0/2690093491 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d8003a20 con 0x7f27e00b2550 2024-01-16T13:52:12.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.738+0000 7f27de7fc700 1 -- 172.21.15.113:0/2690093491 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8003dd0 con 0x7f27e00b2550 2024-01-16T13:52:12.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 -- 172.21.15.113:0/2690093491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00b2550 msgr2=0x7f27e00b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2690093491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00b2550 0x7f27e00b06e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009e50 tx=0x7f27d800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:12.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 -- 172.21.15.113:0/2690093491 shutdown_connections 2024-01-16T13:52:12.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2690093491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00b2550 0x7f27e00b06e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2690093491 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e0163480 0x7f27e00b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 -- 172.21.15.113:0/2690093491 >> 172.21.15.113:0/2690093491 conn(0x7f27e00b57b0 msgr2=0x7f27e00b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.739+0000 7f27e5f39700 1 -- 172.21.15.113:0/2690093491 shutdown_connections 2024-01-16T13:52:12.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.740+0000 7f27e5f39700 1 -- 172.21.15.113:0/2690093491 wait complete. 2024-01-16T13:52:12.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.741+0000 7f27e5f39700 1 Processor -- start 2024-01-16T13:52:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.742+0000 7f27e5f39700 1 -- start start 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27e5f39700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e00b2550 0x7f27e00cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27e5f39700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27e0163480 0x7f27e01664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27e5f39700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 0x7f27e014f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27e5f39700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e014eb00 con 0x7f27e0163860 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27e5f39700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00cb2b0 con 0x7f27e0163480 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27e5f39700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00cb590 con 0x7f27e00b2550 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 0x7f27e014f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 0x7f27e014f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40702/0 (socket says 172.21.15.113:40702) 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.743+0000 7f27dffff700 1 -- 172.21.15.113:0/2268284561 learned_addr learned my addr 172.21.15.113:0/2268284561 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27dffff700 1 -- 172.21.15.113:0/2268284561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e00b2550 msgr2=0x7f27e00cac20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27dffff700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e00b2550 0x7f27e00cac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27dffff700 1 -- 172.21.15.113:0/2268284561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27e0163480 msgr2=0x7f27e01664e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27dffff700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27e0163480 0x7f27e01664e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27dffff700 1 -- 172.21.15.113:0/2268284561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d8009b30 con 0x7f27e0163860 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27dffff700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 0x7f27e014f210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27e00722a0 tx=0x7f27d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.744+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8004380 con 0x7f27e0163860 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.745+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d80044e0 con 0x7f27e0163860 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.745+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8004890 con 0x7f27e0163860 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.745+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e00cb870 con 0x7f27e0163860 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.745+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e0166a20 con 0x7f27e0163860 2024-01-16T13:52:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.746+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e014ef30 con 0x7f27e0163860 2024-01-16T13:52:12.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.747+0000 7fe2f8ca3700 1 Processor -- start 2024-01-16T13:52:12.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.747+0000 7fe2f8ca3700 1 -- start start 2024-01-16T13:52:12.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7fe2f8ca3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f415d6a0 0x7fe2f40b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7fe2f8ca3700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40720e0 con 0x7fe2f4071be0 2024-01-16T13:52:12.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7fe2f8ca3700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40723c0 con 0x7fe2f40b4090 2024-01-16T13:52:12.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7fe2f8ca3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4072d60 con 0x7fe2f415d6a0 2024-01-16T13:52:12.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.748+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f27d8019070 con 0x7f27e0163860 2024-01-16T13:52:12.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7f27c7fff700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f27cc06b420 0x7f27cc06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27d808af50 con 0x7f27e0163860 2024-01-16T13:52:12.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.749+0000 7f27df7fe700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f27cc06b420 0x7f27cc06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.750+0000 7f27df7fe700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f27cc06b420 0x7f27cc06d8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009710 tx=0x7f27d0006c60 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.752+0000 7fe2f1d9b700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe2f40b4090 0x7fe2f40714d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:59178/0 (socket says 172.21.15.113:59178) 2024-01-16T13:52:12.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.752+0000 7fe2f1d9b700 1 -- 172.21.15.113:0/3573416947 learned_addr learned my addr 172.21.15.113:0/3573416947 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:12.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.752+0000 7fe2f259c700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f415d6a0 0x7fe2f40b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.758+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359887092 0 0) 0x7fe2f40720e0 con 0x7fe2f4071be0 2024-01-16T13:52:12.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.758+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2dc0035b0 con 0x7fe2f4071be0 2024-01-16T13:52:12.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.758+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27d8055f50 con 0x7f27e0163860 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560515154 0 0) 0x7fe2f40723c0 con 0x7fe2f40b4090 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f40720e0 con 0x7fe2f40b4090 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252095896 0 0) 0x7fe2dc0035b0 con 0x7fe2f4071be0 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f415d6a0 msgr2=0x7fe2f40b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f415d6a0 0x7fe2f40b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 >> v1:172.21.15.124:6789/0 conn(0x7fe2f40b4090 legacy=0x7fe2f40714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40726a0 con 0x7fe2f4071be0 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fe2ec0018a0 con 0x7fe2f4071be0 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 >> v1:172.21.15.113:6789/0 conn(0x7fe2f4071be0 legacy=0x7fe2f4140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc003bc0 0x7fe2dc003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4072d60 con 0x7fe2dc005210 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c000 con 0x7fe2dc004850 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.760+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c190 con 0x7fe2dc003bc0 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 -- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc003bc0 msgr2=0x7fe2dc003fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc003bc0 0x7fe2dc003fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 -- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc004850 msgr2=0x7fe2dc004cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 -- 172.21.15.113:0/3573416947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40726a0 con 0x7fe2dc005210 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.761+0000 7fe2f2d9d700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2dc00ba10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00a740 tx=0x7fe2dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800210eb0f4373c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.762+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec012640 con 0x7fe2dc005210 2024-01-16T13:52:12.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.762+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec0127a0 con 0x7fe2dc005210 2024-01-16T13:52:12.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.762+0000 7fe2f159a700 1 -- 172.21.15.113:0/3573416947 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec012b10 con 0x7fe2dc005210 2024-01-16T13:52:12.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.763+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/3573416947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 msgr2=0x7fe2dc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.763+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/3573416947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2dc00ba10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00a740 tx=0x7fe2dc005880 comp rx=0 tx=0).stop 2024-01-16T13:52:12.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.763+0000 7fe2f2d9d700 1 -- 172.21.15.113:0/3573416947 reap_dead start 2024-01-16T13:52:12.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.763+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/3573416947 shutdown_connections 2024-01-16T13:52:12.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.763+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/3573416947 >> 172.21.15.113:0/3573416947 conn(0x7fe2f40cfa30 msgr2=0x7fe2f40cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.775+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/3573416947 shutdown_connections 2024-01-16T13:52:12.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.775+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/3573416947 wait complete. 2024-01-16T13:52:12.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.777+0000 7fe2f8ca3700 1 Processor -- start 2024-01-16T13:52:12.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.777+0000 7fe2f8ca3700 1 -- start start 2024-01-16T13:52:12.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.778+0000 7fe2f8ca3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2f4140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.778+0000 7fe2f8ca3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2f40c49b0 0x7fe2f40ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.778+0000 7fe2f8ca3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f40aec10 0x7fe2f40cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.778+0000 7fe2f8ca3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40ca470 con 0x7fe2dc005210 2024-01-16T13:52:12.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.778+0000 7fe2f8ca3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40af090 con 0x7fe2f40c49b0 2024-01-16T13:52:12.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.778+0000 7fe2f8ca3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40c4e30 con 0x7fe2f40aec10 2024-01-16T13:52:12.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2f4140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2f4140680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40720/0 (socket says 172.21.15.113:40720) 2024-01-16T13:52:12.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 -- 172.21.15.113:0/517644802 learned_addr learned my addr 172.21.15.113:0/517644802 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 -- 172.21.15.113:0/517644802 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f40aec10 msgr2=0x7fe2f40cac50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:12.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f40aec10 0x7fe2f40cac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 -- 172.21.15.113:0/517644802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2f40c49b0 msgr2=0x7fe2f40ae6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:12.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2f40c49b0 0x7fe2f40ae6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.780+0000 7fe2f259c700 1 -- 172.21.15.113:0/517644802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40c5950 con 0x7fe2dc005210 2024-01-16T13:52:12.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.781+0000 7fe2f259c700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2f4140680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800af60 tx=0x7fe2e800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.782+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8013c20 con 0x7fe2dc005210 2024-01-16T13:52:12.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.783+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e8013d80 con 0x7fe2dc005210 2024-01-16T13:52:12.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.783+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e801b610 con 0x7fe2dc005210 2024-01-16T13:52:12.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.783+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f4139460 con 0x7fe2dc005210 2024-01-16T13:52:12.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.783+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f40c50b0 con 0x7fe2dc005210 2024-01-16T13:52:12.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.787+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe2e801b770 con 0x7fe2dc005210 2024-01-16T13:52:12.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.787+0000 7fe2e2ffd700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2d406b420 0x7fe2d406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.787+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2e808da00 con 0x7fe2dc005210 2024-01-16T13:52:12.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.788+0000 7fe2f1d9b700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2d406b420 0x7fe2d406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.788+0000 7fe2f1d9b700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2d406b420 0x7fe2d406d8e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40cbb10 tx=0x7fe2e4007cb0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.789+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d80052f0 con 0x7fe2dc005210 2024-01-16T13:52:12.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.799+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2e80588e0 con 0x7fe2dc005210 2024-01-16T13:52:12.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.799+0000 7fc07171f700 1 Processor -- start 2024-01-16T13:52:12.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.799+0000 7fc07171f700 1 -- start start 2024-01-16T13:52:12.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc07171f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c15d6a0 0x7fc06c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc07171f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc06c0b4090 0x7fc06c0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc07171f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc06c071a10 0x7fc06c06cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc07171f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06c072d60 con 0x7fc06c0b4090 2024-01-16T13:52:12.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc07171f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06c06de80 con 0x7fc06c071a10 2024-01-16T13:52:12.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc07171f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c0b4510 con 0x7fc06c15d6a0 2024-01-16T13:52:12.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc06b7fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc06c071a10 0x7fc06c06cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc06b7fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc06c071a10 0x7fc06c06cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57890/0 (socket says 172.21.15.113:57890) 2024-01-16T13:52:12.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.801+0000 7fc06b7fe700 1 -- 172.21.15.113:0/3845516336 learned_addr learned my addr 172.21.15.113:0/3845516336 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06affd700 1 --2- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c15d6a0 0x7fc06c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06a7fc700 1 --2- 172.21.15.113:0/3845516336 >> v2:172.21.15.113:3300/0 conn(0x7fc06c0b4090 0x7fc06c0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06b7fe700 1 -- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c15d6a0 msgr2=0x7fc06c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06b7fe700 1 --2- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c15d6a0 0x7fc06c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06b7fe700 1 -- 172.21.15.113:0/3845516336 >> v2:172.21.15.113:3300/0 conn(0x7fc06c0b4090 msgr2=0x7fc06c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06b7fe700 1 --2- 172.21.15.113:0/3845516336 >> v2:172.21.15.113:3300/0 conn(0x7fc06c0b4090 0x7fc06c0714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06b7fe700 1 -- 172.21.15.113:0/3845516336 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c071e70 con 0x7fc06c071a10 2024-01-16T13:52:12.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7fc06b7fe700 1 --2- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c071a10 0x7fc06c06cac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc06c06edd0 tx=0x7fc060009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3623407eb0325c8b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.802+0000 7f4b332ed700 1 Processor -- start 2024-01-16T13:52:12.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.803+0000 7f4b332ed700 1 -- start start 2024-01-16T13:52:12.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.803+0000 7fc069ffb700 1 -- 172.21.15.113:0/3845516336 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060010a90 con 0x7fc06c071a10 2024-01-16T13:52:12.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.803+0000 7fc069ffb700 1 -- 172.21.15.113:0/3845516336 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc060010bf0 con 0x7fc06c071a10 2024-01-16T13:52:12.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.803+0000 7fc069ffb700 1 -- 172.21.15.113:0/3845516336 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060018440 con 0x7fc06c071a10 2024-01-16T13:52:12.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.804+0000 7f4b332ed700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4b2c0d0520 0x7f4b2c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.804+0000 7f4b332ed700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c15d970 con 0x7f4b2c071be0 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.804+0000 7f4b332ed700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0723c0 con 0x7f4b2c0b3b50 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.804+0000 7f4b332ed700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2c072d60 con 0x7f4b2c0d0520 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.804+0000 7fc07171f700 1 -- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c071a10 msgr2=0x7fc06c06cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.804+0000 7fc07171f700 1 --2- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c071a10 0x7fc06c06cac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc06c06edd0 tx=0x7fc060009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 -- 172.21.15.113:0/3845516336 shutdown_connections 2024-01-16T13:52:12.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 --2- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c071a10 0x7fc06c06cac0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 --2- 172.21.15.113:0/3845516336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c15d6a0 0x7fc06c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 --2- 172.21.15.113:0/3845516336 >> v2:172.21.15.113:3300/0 conn(0x7fc06c0b4090 0x7fc06c0714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 -- 172.21.15.113:0/3845516336 >> 172.21.15.113:0/3845516336 conn(0x7fc06c0cfa30 msgr2=0x7fc06c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 -- 172.21.15.113:0/3845516336 shutdown_connections 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7f4b3188a700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f4b2c071be0 0x7f4b2c140fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:52118/0 (socket says 172.21.15.113:52118) 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7f4b3188a700 1 -- 172.21.15.113:0/1944874912 learned_addr learned my addr 172.21.15.113:0/1944874912 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:12.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.805+0000 7fc07171f700 1 -- 172.21.15.113:0/3845516336 wait complete. 2024-01-16T13:52:12.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.806+0000 7f4b31089700 1 --2- 172.21.15.113:0/1944874912 >> v2:172.21.15.124:3300/0 conn(0x7f4b2c0d0520 0x7f4b2c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.806+0000 7f4b23fff700 1 -- 172.21.15.113:0/1944874912 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265114964 0 0) 0x7f4b2c15d970 con 0x7f4b2c071be0 2024-01-16T13:52:12.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.806+0000 7f4b23fff700 1 -- 172.21.15.113:0/1944874912 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b180035b0 con 0x7f4b2c071be0 2024-01-16T13:52:12.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.806+0000 7f4b31089700 1 -- 172.21.15.113:0/1944874912 >> v1:172.21.15.124:6789/0 conn(0x7f4b2c0b3b50 legacy=0x7f4b2c0714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.806+0000 7f4b31089700 1 -- 172.21.15.113:0/1944874912 >> v1:172.21.15.113:6789/0 conn(0x7f4b2c071be0 legacy=0x7f4b2c140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.806+0000 7f4b31089700 1 -- 172.21.15.113:0/1944874912 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c160ba0 con 0x7f4b2c0d0520 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.807+0000 7f4b31089700 1 --2- 172.21.15.113:0/1944874912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c15d430 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009d90 tx=0x7f4b2400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92048ac588558ee9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.807+0000 7fc07171f700 1 Processor -- start 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.807+0000 7f4b23fff700 1 -- 172.21.15.113:0/1944874912 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2400e070 con 0x7f4b2c0d0520 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.807+0000 7f4b23fff700 1 -- 172.21.15.113:0/1944874912 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2400baa0 con 0x7f4b2c0d0520 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.807+0000 7f4b23fff700 1 -- 172.21.15.113:0/1944874912 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2400be50 con 0x7f4b2c0d0520 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7f4b332ed700 1 -- 172.21.15.113:0/1944874912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0d0520 msgr2=0x7f4b2c15d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7f4b332ed700 1 --2- 172.21.15.113:0/1944874912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c15d430 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009d90 tx=0x7f4b2400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:12.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.807+0000 7fc07171f700 1 -- start start 2024-01-16T13:52:12.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc07171f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c071a10 0x7fc06c158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc07171f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c0b4090 0x7fc06c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc07171f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 0x7fc06c1539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc07171f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c155d90 con 0x7fc06c15d6a0 2024-01-16T13:52:12.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc07171f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c156070 con 0x7fc06c0b4090 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc07171f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c156350 con 0x7fc06c071a10 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.808+0000 7fc06b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 0x7fc06c1539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 0x7fc06c1539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40740/0 (socket says 172.21.15.113:40740) 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 -- 172.21.15.113:0/53045162 learned_addr learned my addr 172.21.15.113:0/53045162 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06affd700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c071a10 0x7fc06c158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 -- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c071a10 msgr2=0x7fc06c158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c071a10 0x7fc06c158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 -- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c0b4090 msgr2=0x7fc06c155540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c0b4090 0x7fc06c155540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 -- 172.21.15.113:0/53045162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc060009550 con 0x7fc06c15d6a0 2024-01-16T13:52:12.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.809+0000 7fc06b7fe700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 0x7fc06c1539a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc060009f20 tx=0x7fc060004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.810+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060010a50 con 0x7fc06c15d6a0 2024-01-16T13:52:12.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.810+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0600036a0 con 0x7fc06c15d6a0 2024-01-16T13:52:12.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.810+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060003a10 con 0x7fc06c15d6a0 2024-01-16T13:52:12.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.810+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06c156630 con 0x7fc06c15d6a0 2024-01-16T13:52:12.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.810+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c155a80 con 0x7fc06c15d6a0 2024-01-16T13:52:12.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.812+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06c0751e0 con 0x7fc06c15d6a0 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.813+0000 7f4b332ed700 1 -- 172.21.15.113:0/1944874912 shutdown_connections 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.813+0000 7f4b332ed700 1 --2- 172.21.15.113:0/1944874912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c15d430 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.813+0000 7f4b332ed700 1 -- 172.21.15.113:0/1944874912 >> 172.21.15.113:0/1944874912 conn(0x7f4b2c0cfa30 msgr2=0x7f4b2c0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.813+0000 7f4b332ed700 1 -- 172.21.15.113:0/1944874912 shutdown_connections 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.813+0000 7f4b332ed700 1 -- 172.21.15.113:0/1944874912 wait complete. 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.815+0000 7f4b332ed700 1 Processor -- start 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.815+0000 7f4b332ed700 1 -- start start 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7f4b332ed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b2c071be0 0x7f4b2c0aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7f4b332ed700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0b3b50 0x7f4b2c15ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7f4b332ed700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7f4b332ed700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c16bed0 con 0x7f4b2c0d0520 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7f4b332ed700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1409b0 con 0x7f4b2c0b3b50 2024-01-16T13:52:12.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7f4b332ed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1f9700 con 0x7f4b2c071be0 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.818+0000 7f4b31089700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b2c071be0 0x7f4b2c0aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.818+0000 7f4b30888700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0b3b50 0x7f4b2c15ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc060003b70 con 0x7fc06c15d6a0 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7fc0537fe700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc05406b5b0 0x7fc05406da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.817+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc06008ac10 con 0x7fc06c15d6a0 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.818+0000 7f4b3188a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.818+0000 7f4b3188a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c14fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40746/0 (socket says 172.21.15.113:40746) 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.818+0000 7f4b3188a700 1 -- 172.21.15.113:0/3545073202 learned_addr learned my addr 172.21.15.113:0/3545073202 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.818+0000 7fc06affd700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc05406b5b0 0x7fc05406da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.819+0000 7f4b3188a700 1 -- 172.21.15.113:0/3545073202 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b2c071be0 msgr2=0x7f4b2c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.819+0000 7f4b3188a700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b2c071be0 0x7f4b2c0aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.819+0000 7f4b3188a700 1 -- 172.21.15.113:0/3545073202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0b3b50 msgr2=0x7f4b2c15ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.819+0000 7f4b3188a700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0b3b50 0x7f4b2c15ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.819+0000 7f4b3188a700 1 -- 172.21.15.113:0/3545073202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24009a70 con 0x7f4b2c0d0520 2024-01-16T13:52:12.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.819+0000 7fc06affd700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc05406b5b0 0x7fc05406da70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc06400afd0 tx=0x7fc06400d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.820+0000 7f4b3188a700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c14fc20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800c4b0 tx=0x7f4b2800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.822+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc060055ba0 con 0x7fc06c15d6a0 2024-01-16T13:52:12.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.822+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28004d40 con 0x7f4b2c0d0520 2024-01-16T13:52:12.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.822+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b280070d0 con 0x7f4b2c0d0520 2024-01-16T13:52:12.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.822+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28004ea0 con 0x7f4b2c0d0520 2024-01-16T13:52:12.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.823+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c160520 con 0x7f4b2c0d0520 2024-01-16T13:52:12.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.823+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2c15f610 con 0x7f4b2c0d0520 2024-01-16T13:52:12.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.827+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4b28007440 con 0x7f4b2c0d0520 2024-01-16T13:52:12.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.828+0000 7f4b21ffb700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4b1806b880 0x7f4b1806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.828+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b28089770 con 0x7f4b2c0d0520 2024-01-16T13:52:12.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.828+0000 7f4b31089700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4b1806b880 0x7f4b1806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.829+0000 7f4b31089700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4b1806b880 0x7f4b1806dd40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4b24000f80 tx=0x7f4b24011000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.831+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2c0d0300 con 0x7f4b2c0d0520 2024-01-16T13:52:12.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.840+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b28054650 con 0x7f4b2c0d0520 2024-01-16T13:52:12.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.868+0000 7fbeb276b700 1 Processor -- start 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.868+0000 7fbeb276b700 1 -- start start 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.870+0000 7fbeb276b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbeac0b3b50 0x7fbeac0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.870+0000 7fbeb276b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 0x7fbeac06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.870+0000 7fbeb276b700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac06ced0 con 0x7fbeac163a80 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.870+0000 7fbeb276b700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeac06da70 con 0x7fbeac0b3b50 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.870+0000 7fbeb276b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0b4470 con 0x7fbeac06d4d0 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.871+0000 7fbeab7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 0x7fbeac06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.871+0000 7fbeab7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 0x7fbeac06eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:36916/0 (socket says 172.21.15.113:36916) 2024-01-16T13:52:12.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.871+0000 7fbeab7fe700 1 -- 172.21.15.113:0/976638556 learned_addr learned my addr 172.21.15.113:0/976638556 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.871+0000 7fbeabfff700 1 --2- 172.21.15.113:0/976638556 >> v2:172.21.15.124:3300/0 conn(0x7fbeac0b3b50 0x7fbeac0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.872+0000 7fbeab7fe700 1 -- 172.21.15.113:0/976638556 >> v2:172.21.15.124:3300/0 conn(0x7fbeac0b3b50 msgr2=0x7fbeac0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.872+0000 7fbeab7fe700 1 --2- 172.21.15.113:0/976638556 >> v2:172.21.15.124:3300/0 conn(0x7fbeac0b3b50 0x7fbeac0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.872+0000 7fbeab7fe700 1 -- 172.21.15.113:0/976638556 >> v1:172.21.15.113:6789/0 conn(0x7fbeac163a80 legacy=0x7fbeac163f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.872+0000 7fbeab7fe700 1 -- 172.21.15.113:0/976638556 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac1634e0 con 0x7fbeac06d4d0 2024-01-16T13:52:12.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.873+0000 7fbeab7fe700 1 --2- 172.21.15.113:0/976638556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 0x7fbeac06eee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbea00098f0 tx=0x7fbea0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=495dee45c615c271 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.873+0000 7fbeaaffd700 1 -- 172.21.15.113:0/976638556 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0011830 con 0x7fbeac06d4d0 2024-01-16T13:52:12.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.873+0000 7fbeaaffd700 1 -- 172.21.15.113:0/976638556 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea0011990 con 0x7fbeac06d4d0 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.873+0000 7fbeaaffd700 1 -- 172.21.15.113:0/976638556 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea0011d20 con 0x7fbeac06d4d0 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.874+0000 7fbeb276b700 1 -- 172.21.15.113:0/976638556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 msgr2=0x7fbeac06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.874+0000 7fbeb276b700 1 --2- 172.21.15.113:0/976638556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 0x7fbeac06eee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbea00098f0 tx=0x7fbea0009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.874+0000 7fbeb276b700 1 -- 172.21.15.113:0/976638556 shutdown_connections 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.874+0000 7fbeb276b700 1 --2- 172.21.15.113:0/976638556 >> v2:172.21.15.124:3300/0 conn(0x7fbeac0b3b50 0x7fbeac0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.874+0000 7fbeb276b700 1 --2- 172.21.15.113:0/976638556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac06d4d0 0x7fbeac06eee0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.874+0000 7fbeb276b700 1 -- 172.21.15.113:0/976638556 >> 172.21.15.113:0/976638556 conn(0x7fbeac0cfa30 msgr2=0x7fbeac0cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.875+0000 7fbeb276b700 1 -- 172.21.15.113:0/976638556 shutdown_connections 2024-01-16T13:52:12.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.875+0000 7fbeb276b700 1 -- 172.21.15.113:0/976638556 wait complete. 2024-01-16T13:52:12.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.876+0000 7fbeb276b700 1 Processor -- start 2024-01-16T13:52:12.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.877+0000 7fbeb276b700 1 -- start start 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeb276b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbeac06d4d0 0x7fbeac0aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeb276b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeac0b3b50 0x7fbeac072310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeb276b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac163a80 0x7fbeac072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeb276b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac15f800 con 0x7fbeac0b3b50 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeb276b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac15f1f0 con 0x7fbeac06d4d0 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeb276b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0ab820 con 0x7fbeac163a80 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeab7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeac0b3b50 0x7fbeac072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeab7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeac0b3b50 0x7fbeac072310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40752/0 (socket says 172.21.15.113:40752) 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.878+0000 7fbeab7fe700 1 -- 172.21.15.113:0/1042033695 learned_addr learned my addr 172.21.15.113:0/1042033695 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeb0d08700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac163a80 0x7fbeac072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeabfff700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbeac06d4d0 0x7fbeac0aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeb0d08700 1 -- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbeac06d4d0 msgr2=0x7fbeac0aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeb0d08700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbeac06d4d0 0x7fbeac0aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeb0d08700 1 -- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeac0b3b50 msgr2=0x7fbeac072310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeb0d08700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeac0b3b50 0x7fbeac072310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.879+0000 7fbeb0d08700 1 -- 172.21.15.113:0/1042033695 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0009550 con 0x7fbeac163a80 2024-01-16T13:52:12.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.883+0000 7fbeb0d08700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac163a80 0x7fbeac072850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbea400caa0 tx=0x7fbea400cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.883+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea4003c00 con 0x7fbeac163a80 2024-01-16T13:52:12.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.883+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea4003d60 con 0x7fbeac163a80 2024-01-16T13:52:12.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.883+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea400a9c0 con 0x7fbeac163a80 2024-01-16T13:52:12.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.883+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeac15f490 con 0x7fbeac163a80 2024-01-16T13:52:12.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.884+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeac161390 con 0x7fbeac163a80 2024-01-16T13:52:12.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.886+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8c0052f0 con 0x7fbeac163a80 2024-01-16T13:52:12.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.888+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbea400ab20 con 0x7fbeac163a80 2024-01-16T13:52:12.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.889+0000 7fbea8ff9700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe9806b420 0x7fbe9806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.889+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbea408a450 con 0x7fbeac163a80 2024-01-16T13:52:12.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.889+0000 7fbeabfff700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe9806b420 0x7fbe9806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.890+0000 7fbeabfff700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe9806b420 0x7fbe9806d8e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009be0 tx=0x7fbe9c009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.892+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbea4055360 con 0x7fbeac163a80 2024-01-16T13:52:12.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.930+0000 7f42d052f700 1 Processor -- start 2024-01-16T13:52:12.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.930+0000 7f42d052f700 1 -- start start 2024-01-16T13:52:12.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.931+0000 7f42d052f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f42c815d6a0 0x7f42c80b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.931+0000 7f42d052f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f42c80b4090 0x7f42c80714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.931+0000 7f42d052f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c80720e0 con 0x7f42c8071a10 2024-01-16T13:52:12.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.931+0000 7f42d052f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c8072d60 con 0x7f42c80b4090 2024-01-16T13:52:12.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.931+0000 7f42d052f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c806de80 con 0x7f42c815d6a0 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.932+0000 7f42ce2cb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f42c815d6a0 0x7f42c80b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.932+0000 7f42ce2cb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f42c815d6a0 0x7f42c80b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:57956/0 (socket says 172.21.15.113:57956) 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.932+0000 7f42ce2cb700 1 -- 172.21.15.113:0/144310516 learned_addr learned my addr 172.21.15.113:0/144310516 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42cdaca700 1 --2- 172.21.15.113:0/144310516 >> v2:172.21.15.113:3300/0 conn(0x7f42c80b4090 0x7f42c80714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42ce2cb700 1 -- 172.21.15.113:0/144310516 >> v2:172.21.15.113:3300/0 conn(0x7f42c80b4090 msgr2=0x7f42c80714d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42ce2cb700 1 --2- 172.21.15.113:0/144310516 >> v2:172.21.15.113:3300/0 conn(0x7f42c80b4090 0x7f42c80714d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42ce2cb700 1 -- 172.21.15.113:0/144310516 >> v1:172.21.15.113:6789/0 conn(0x7f42c8071a10 legacy=0x7f42c806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42ce2cb700 1 -- 172.21.15.113:0/144310516 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c80723c0 con 0x7f42c815d6a0 2024-01-16T13:52:12.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42ce2cb700 1 --2- 172.21.15.113:0/144310516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 0x7f42c80b3b50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f42c80cfa00 tx=0x7f42c000d650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9d83b3bbed453b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42cd2c9700 1 -- 172.21.15.113:0/144310516 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0010070 con 0x7f42c815d6a0 2024-01-16T13:52:12.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42cd2c9700 1 -- 172.21.15.113:0/144310516 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c000dce0 con 0x7f42c815d6a0 2024-01-16T13:52:12.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.933+0000 7f42cd2c9700 1 -- 172.21.15.113:0/144310516 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0007d30 con 0x7f42c815d6a0 2024-01-16T13:52:12.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.934+0000 7f42d052f700 1 -- 172.21.15.113:0/144310516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 msgr2=0x7f42c80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.934+0000 7f42d052f700 1 --2- 172.21.15.113:0/144310516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 0x7f42c80b3b50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f42c80cfa00 tx=0x7f42c000d650 comp rx=0 tx=0).stop 2024-01-16T13:52:12.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.936+0000 7f42d052f700 1 -- 172.21.15.113:0/144310516 shutdown_connections 2024-01-16T13:52:12.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.936+0000 7f42d052f700 1 --2- 172.21.15.113:0/144310516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 0x7f42c80b3b50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.936+0000 7f42d052f700 1 --2- 172.21.15.113:0/144310516 >> v2:172.21.15.113:3300/0 conn(0x7f42c80b4090 0x7f42c80714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.936+0000 7f42d052f700 1 -- 172.21.15.113:0/144310516 >> 172.21.15.113:0/144310516 conn(0x7f42c80cfa30 msgr2=0x7f42c80cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:12.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.936+0000 7f42d052f700 1 -- 172.21.15.113:0/144310516 shutdown_connections 2024-01-16T13:52:12.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.937+0000 7f42d052f700 1 -- 172.21.15.113:0/144310516 wait complete. 2024-01-16T13:52:12.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.938+0000 7f42d052f700 1 Processor -- start 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.939+0000 7f42d052f700 1 -- start start 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.940+0000 7f42d052f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 0x7f42c81f93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.940+0000 7f42d052f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42c80b4090 0x7f42c80aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.940+0000 7f42d052f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 0x7f42c80b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.940+0000 7f42d052f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81f8f10 con 0x7f42c8071a10 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.940+0000 7f42d052f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c813cd30 con 0x7f42c815d6a0 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.940+0000 7f42d052f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c813ca50 con 0x7f42c80b4090 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 0x7f42c81f93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 0x7f42c81f93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40762/0 (socket says 172.21.15.113:40762) 2024-01-16T13:52:12.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 -- 172.21.15.113:0/2456591398 learned_addr learned my addr 172.21.15.113:0/2456591398 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42cdaca700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42c80b4090 0x7f42c80aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 -- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42c80b4090 msgr2=0x7f42c80aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42c80b4090 0x7f42c80aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 -- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 msgr2=0x7f42c80b2c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 0x7f42c80b2c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.941+0000 7f42ce2cb700 1 -- 172.21.15.113:0/2456591398 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c000d050 con 0x7f42c8071a10 2024-01-16T13:52:12.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.942+0000 7f42ce2cb700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 0x7f42c81f93a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f42c80cc240 tx=0x7f42c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.943+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0004450 con 0x7f42c8071a10 2024-01-16T13:52:12.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.943+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c00045b0 con 0x7f42c8071a10 2024-01-16T13:52:12.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.943+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0004940 con 0x7f42c8071a10 2024-01-16T13:52:12.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.943+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c81f88c0 con 0x7f42c8071a10 2024-01-16T13:52:12.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.943+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c80b3140 con 0x7f42c8071a10 2024-01-16T13:52:12.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.946+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f42c001d070 con 0x7f42c8071a10 2024-01-16T13:52:12.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.947+0000 7f42beffd700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f42b406b5b0 0x7f42b406da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:12.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.947+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42c008af80 con 0x7f42c8071a10 2024-01-16T13:52:12.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.948+0000 7f42cdaca700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f42b406b5b0 0x7f42b406da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:12.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.948+0000 7f42cdaca700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f42b406b5b0 0x7f42b406da70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f42c8153a50 tx=0x7f42c4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:12.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.952+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac0052f0 con 0x7f42c8071a10 2024-01-16T13:52:12.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:12.962+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42c0055f80 con 0x7f42c8071a10 2024-01-16T13:52:13.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.086+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd1580ced10 con 0x7fd1580b2fc0 2024-01-16T13:52:13.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.087+0000 7fd14dffb700 1 -- 172.21.15.113:0/4124231166 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd154053d40 con 0x7fd1580b2fc0 2024-01-16T13:52:13.072 INFO:teuthology.orchestra.run.smithi113.stdout:55834574852 2024-01-16T13:52:13.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.090+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd13c06b420 msgr2=0x7fd13c06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.090+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd13c06b420 0x7fd13c06d8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd148008000 tx=0x7fd148008030 comp rx=0 tx=0).stop 2024-01-16T13:52:13.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.090+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1580b2fc0 msgr2=0x7fd1581f27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.090+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1580b2fc0 0x7fd1581f27a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c500 tx=0x7fd15400c810 comp rx=0 tx=0).stop 2024-01-16T13:52:13.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.091+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 shutdown_connections 2024-01-16T13:52:13.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.091+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd13c06b420 0x7fd13c06d8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.091+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1581f87c0 0x7fd1580d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.091+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1580b2fc0 0x7fd1581f27a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.091+0000 7fd15f7a7700 1 --2- 172.21.15.113:0/4124231166 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd15815dfc0 0x7fd158153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.091+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 >> 172.21.15.113:0/4124231166 conn(0x7fd15806b410 msgr2=0x7fd1580b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.092+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 shutdown_connections 2024-01-16T13:52:13.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.092+0000 7fd15f7a7700 1 -- 172.21.15.113:0/4124231166 wait complete. 2024-01-16T13:52:13.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-01-16T13:52:13.092 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.125+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3534143cc0 con 0x7f353415d6a0 2024-01-16T13:52:13.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.126+0000 7f351bfff700 1 -- 172.21.15.113:0/3909356736 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3528057d80 con 0x7f353415d6a0 2024-01-16T13:52:13.109 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f352006b380 msgr2=0x7f352006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f352006b380 0x7f352006d840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f352400d370 tx=0x7f3524012000 comp rx=0 tx=0).stop 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 msgr2=0x7f35340cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 0x7f35340cc420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f352800b470 tx=0x7f352800b830 comp rx=0 tx=0).stop 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 shutdown_connections 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f352006b380 0x7f352006d840 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f352400d370 tx=0x7f3524012000 comp rx=0 tx=0).stop 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35340cdc10 0x7f35341f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3534071be0 0x7f35340aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.129+0000 7f353a20f700 1 --2- 172.21.15.113:0/3909356736 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f353415d6a0 0x7f35340cc420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.130+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 >> 172.21.15.113:0/3909356736 conn(0x7f35340cfa30 msgr2=0x7f35340cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.130+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 shutdown_connections 2024-01-16T13:52:13.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.130+0000 7f353a20f700 1 -- 172.21.15.113:0/3909356736 wait complete. 2024-01-16T13:52:13.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-01-16T13:52:13.131 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.171+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f27e014ef30 con 0x7f27e0163860 2024-01-16T13:52:13.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.171+0000 7f27c7fff700 1 -- 172.21.15.113:0/2268284561 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f27e014ef30 con 0x7f27e0163860 2024-01-16T13:52:13.153 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f27cc06b420 msgr2=0x7f27cc06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f27cc06b420 0x7f27cc06d8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009710 tx=0x7f27d0006c60 comp rx=0 tx=0).stop 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 msgr2=0x7f27e014f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 0x7f27e014f210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27e00722a0 tx=0x7f27d8004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 shutdown_connections 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f27cc06b420 0x7f27cc06d8e0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009710 tx=0x7f27d0006c60 comp rx=0 tx=0).stop 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e0163860 0x7f27e014f210 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f27e00b2550 0x7f27e00cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 --2- 172.21.15.113:0/2268284561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27e0163480 0x7f27e01664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.175+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 >> 172.21.15.113:0/2268284561 conn(0x7f27e00b57b0 msgr2=0x7f27e0160ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.176+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 shutdown_connections 2024-01-16T13:52:13.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.178+0000 7f27e5f39700 1 -- 172.21.15.113:0/2268284561 wait complete. 2024-01-16T13:52:13.177 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-01-16T13:52:13.177 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.201+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe2d80055a0 con 0x7fe2dc005210 2024-01-16T13:52:13.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.202+0000 7fe2e2ffd700 1 -- 172.21.15.113:0/517644802 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2e805bf00 con 0x7fe2dc005210 2024-01-16T13:52:13.184 INFO:teuthology.orchestra.run.smithi113.stdout:55834574854 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2d406b420 msgr2=0x7fe2d406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2d406b420 0x7fe2d406d8e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40cbb10 tx=0x7fe2e4007cb0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 msgr2=0x7fe2f4140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2f4140680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800af60 tx=0x7fe2e800af90 comp rx=0 tx=0).stop 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 shutdown_connections 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2d406b420 0x7fe2d406d8e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2f40aec10 0x7fe2f40cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc005210 0x7fe2f4140680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 --2- 172.21.15.113:0/517644802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2f40c49b0 0x7fe2f40ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 >> 172.21.15.113:0/517644802 conn(0x7fe2f40cfa30 msgr2=0x7fe2f40cfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.204+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 shutdown_connections 2024-01-16T13:52:13.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.205+0000 7fe2f8ca3700 1 -- 172.21.15.113:0/517644802 wait complete. 2024-01-16T13:52:13.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.206+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbe8c005870 con 0x7fbeac163a80 2024-01-16T13:52:13.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.207+0000 7fbea8ff9700 1 -- 172.21.15.113:0/1042033695 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbea4058980 con 0x7fbeac163a80 2024-01-16T13:52:13.189 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe9806b420 msgr2=0x7fbe9806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe9806b420 0x7fbe9806d8e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009be0 tx=0x7fbe9c009210 comp rx=0 tx=0).stop 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac163a80 msgr2=0x7fbeac072850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac163a80 0x7fbeac072850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbea400caa0 tx=0x7fbea400cdb0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 shutdown_connections 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe9806b420 0x7fbe9806d8e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbeac163a80 0x7fbeac072850 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbeac06d4d0 0x7fbeac0aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 --2- 172.21.15.113:0/1042033695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeac0b3b50 0x7fbeac072310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.209+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 >> 172.21.15.113:0/1042033695 conn(0x7fbeac0cfa30 msgr2=0x7fbeac070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.210+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 shutdown_connections 2024-01-16T13:52:13.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.210+0000 7fbeb276b700 1 -- 172.21.15.113:0/1042033695 wait complete. 2024-01-16T13:52:13.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-01-16T13:52:13.200 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-01-16T13:52:13.205 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.226+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc06c0b0560 con 0x7fc06c15d6a0 2024-01-16T13:52:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.227+0000 7fc0537fe700 1 -- 172.21.15.113:0/53045162 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc060028020 con 0x7fc06c15d6a0 2024-01-16T13:52:13.209 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc05406b5b0 msgr2=0x7fc05406da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc05406b5b0 0x7fc05406da70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc06400afd0 tx=0x7fc06400d040 comp rx=0 tx=0).stop 2024-01-16T13:52:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 msgr2=0x7fc06c1539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 0x7fc06c1539a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc060009f20 tx=0x7fc060004750 comp rx=0 tx=0).stop 2024-01-16T13:52:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 shutdown_connections 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc05406b5b0 0x7fc05406da70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc06c15d6a0 0x7fc06c1539a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc06c071a10 0x7fc06c158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 --2- 172.21.15.113:0/53045162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc06c0b4090 0x7fc06c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 >> 172.21.15.113:0/53045162 conn(0x7fc06c0cfa30 msgr2=0x7fc06c15c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 shutdown_connections 2024-01-16T13:52:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.230+0000 7fc07171f700 1 -- 172.21.15.113:0/53045162 wait complete. 2024-01-16T13:52:13.229 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2024-01-16T13:52:13.229 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.249+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4b2c0751e0 con 0x7f4b2c0d0520 2024-01-16T13:52:13.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.249+0000 7f4b21ffb700 1 -- 172.21.15.113:0/3545073202 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4b28057c70 con 0x7f4b2c0d0520 2024-01-16T13:52:13.231 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:13.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4b1806b880 msgr2=0x7f4b1806dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4b1806b880 0x7f4b1806dd40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4b24000f80 tx=0x7f4b24011000 comp rx=0 tx=0).stop 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 msgr2=0x7f4b2c14fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c14fc20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800c4b0 tx=0x7f4b2800c870 comp rx=0 tx=0).stop 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 shutdown_connections 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4b1806b880 0x7f4b1806dd40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b2c0d0520 0x7f4b2c14fc20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b2c071be0 0x7f4b2c0aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 --2- 172.21.15.113:0/3545073202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b2c0b3b50 0x7f4b2c15ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.251+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 >> 172.21.15.113:0/3545073202 conn(0x7f4b2c0cfa30 msgr2=0x7f4b2c1432b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.252+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 shutdown_connections 2024-01-16T13:52:13.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.252+0000 7f4b332ed700 1 -- 172.21.15.113:0/3545073202 wait complete. 2024-01-16T13:52:13.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.262+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f42b00051a0 con 0x7f42c8071a10 2024-01-16T13:52:13.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.263+0000 7f42beffd700 1 -- 172.21.15.113:0/2456591398 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42c001b030 con 0x7f42c8071a10 2024-01-16T13:52:13.245 INFO:teuthology.orchestra.run.smithi113.stdout:55834574853 2024-01-16T13:52:13.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2024-01-16T13:52:13.246 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f42b406b5b0 msgr2=0x7f42b406da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f42b406b5b0 0x7f42b406da70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f42c8153a50 tx=0x7f42c4008040 comp rx=0 tx=0).stop 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 msgr2=0x7f42c81f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 0x7f42c81f93a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f42c80cc240 tx=0x7f42c0004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 shutdown_connections 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f42b406b5b0 0x7f42b406da70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42c815d6a0 0x7f42c80b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42c8071a10 0x7f42c81f93a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 --2- 172.21.15.113:0/2456591398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42c80b4090 0x7f42c80aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 >> 172.21.15.113:0/2456591398 conn(0x7f42c80cfa30 msgr2=0x7f42c81410d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 shutdown_connections 2024-01-16T13:52:13.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.265+0000 7f42d052f700 1 -- 172.21.15.113:0/2456591398 wait complete. 2024-01-16T13:52:13.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-01-16T13:52:13.259 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:13.259 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T13:52:13.259 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:52:13.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.438+0000 7f04b7762700 1 Processor -- start 2024-01-16T13:52:13.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.439+0000 7f04b7762700 1 -- start start 2024-01-16T13:52:13.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.439+0000 7f04b7762700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04b0147780 0x7f04b015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.439+0000 7f04b7762700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.439+0000 7f04b7762700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b015eb00 con 0x7f04b01473a0 2024-01-16T13:52:13.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.439+0000 7f04b7762700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b015ede0 con 0x7f04b0147780 2024-01-16T13:52:13.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.439+0000 7f04b7762700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b015f0c0 con 0x7f04b006dc20 2024-01-16T13:52:13.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04b0147780 0x7f04b015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f04b0147780 0x7f04b015e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48212/0 (socket says 172.21.15.113:48212) 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 -- 172.21.15.113:0/33928986 learned_addr learned my addr 172.21.15.113:0/33928986 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b5cff700 1 --2- 172.21.15.113:0/33928986 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 -- 172.21.15.113:0/33928986 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 msgr2=0x7f04b0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 --2- 172.21.15.113:0/33928986 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b0073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 -- 172.21.15.113:0/33928986 >> v1:172.21.15.124:6789/0 conn(0x7f04b01473a0 legacy=0x7f04b006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 -- 172.21.15.113:0/33928986 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b015f3a0 con 0x7f04b0147780 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04b4cfd700 1 --2- 172.21.15.113:0/33928986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b015e4b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098f0 tx=0x7f04ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb1ba7087cae9e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04a7fff700 1 -- 172.21.15.113:0/33928986 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac011830 con 0x7f04b0147780 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04a7fff700 1 -- 172.21.15.113:0/33928986 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac011990 con 0x7f04b0147780 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.440+0000 7f04a7fff700 1 -- 172.21.15.113:0/33928986 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac011d20 con 0x7f04b0147780 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 -- 172.21.15.113:0/33928986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 msgr2=0x7f04b015e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 --2- 172.21.15.113:0/33928986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b015e4b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098f0 tx=0x7f04ac009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:13.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 -- 172.21.15.113:0/33928986 shutdown_connections 2024-01-16T13:52:13.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 --2- 172.21.15.113:0/33928986 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 --2- 172.21.15.113:0/33928986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b015e4b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 -- 172.21.15.113:0/33928986 >> 172.21.15.113:0/33928986 conn(0x7f04b00b3130 msgr2=0x7f04b00d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 -- 172.21.15.113:0/33928986 shutdown_connections 2024-01-16T13:52:13.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.441+0000 7f04b7762700 1 -- 172.21.15.113:0/33928986 wait complete. 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.442+0000 7f04b7762700 1 Processor -- start 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.442+0000 7f04b7762700 1 -- start start 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b00cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f04b01473a0 0x7f04b015d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b0145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00cfd60 con 0x7f04b0147780 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00d0040 con 0x7f04b01473a0 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00d0320 con 0x7f04b006dc20 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b0145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b0145c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48226/0 (socket says 172.21.15.113:48226) 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 -- 172.21.15.113:0/3359060270 learned_addr learned my addr 172.21.15.113:0/3359060270 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:13.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b54fe700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b00cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b4cfd700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f04b01473a0 0x7f04b015d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 -- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 msgr2=0x7f04b00cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b00cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 -- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f04b01473a0 msgr2=0x7f04b015d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f04b01473a0 0x7f04b015d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 -- 172.21.15.113:0/3359060270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac009550 con 0x7f04b0147780 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b5cff700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b0145c10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04a800efc0 tx=0x7f04a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8009ea0 con 0x7f04b0147780 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a800f040 con 0x7f04b0147780 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a800ced0 con 0x7f04b0147780 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.443+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b00ad870 con 0x7f04b0147780 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.444+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0146150 con 0x7f04b0147780 2024-01-16T13:52:13.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.444+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0067f60 con 0x7f04b0147780 2024-01-16T13:52:13.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.445+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f04a8014480 con 0x7f04b0147780 2024-01-16T13:52:13.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.445+0000 7f04a5ffb700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f049406b350 0x7f049406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.445+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04a808a2d0 con 0x7f04b0147780 2024-01-16T13:52:13.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.445+0000 7f04b54fe700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f049406b350 0x7f049406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.446+0000 7f04b54fe700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f049406b350 0x7f049406d810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04b01406e0 tx=0x7f04a0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:13.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.450+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04a80551d0 con 0x7f04b0147780 2024-01-16T13:52:13.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.636+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04b00744d0 con 0x7f049406b350 2024-01-16T13:52:13.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.643+0000 7f04a5ffb700 1 -- 172.21.15.113:0/3359060270 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264606 (secure 0 0 0) 0x7f04a0001580 con 0x7f049406b350 2024-01-16T13:52:13.628 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:13.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f049406b350 msgr2=0x7f049406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f049406b350 0x7f049406d810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04b01406e0 tx=0x7f04a0009040 comp rx=0 tx=0).stop 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 msgr2=0x7f04b0145c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b0145c10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04a800efc0 tx=0x7f04a800c3a0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 shutdown_connections 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f049406b350 0x7f049406d810 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.646+0000 7f04b7762700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04b0147780 0x7f04b0145c10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.647+0000 7f04b7762700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f04b006dc20 0x7f04b00cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.647+0000 7f04b7762700 1 --2- 172.21.15.113:0/3359060270 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f04b01473a0 0x7f04b015d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.647+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 >> 172.21.15.113:0/3359060270 conn(0x7f04b00b3130 msgr2=0x7f04b015fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.647+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 shutdown_connections 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.647+0000 7f04b7762700 1 -- 172.21.15.113:0/3359060270 wait complete. 2024-01-16T13:52:13.631 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:52:13.640 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-16T13:52:12.403093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001562"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383364+0000","last_change":"2024-01-16T13:52:04.933545+0000","last_active":"2024-01-16T13:52:06.383364+0000","last_peered":"2024-01-16T13:52:06.383364+0000","last_clean":"2024-01-16T13:52:06.383364+0000","last_became_active":"2024-01-16T13:52:04.932314+0000","last_became_peered":"2024-01-16T13:52:04.932314+0000","last_unstale":"2024-01-16T13:52:06.383364+0000","last_undegraded":"2024-01-16T13:52:06.383364+0000","last_fullsized":"2024-01-16T13:52:06.383364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.917385+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.916958+0000","last_became_peered":"2024-01-16T13:52:04.916958+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382533+0000","last_change":"2024-01-16T13:52:04.917514+0000","last_active":"2024-01-16T13:52:06.382533+0000","last_peered":"2024-01-16T13:52:06.382533+0000","last_clean":"2024-01-16T13:52:06.382533+0000","last_became_active":"2024-01-16T13:52:04.916822+0000","last_became_peered":"2024-01-16T13:52:04.916822+0000","last_unstale":"2024-01-16T13:52:06.382533+0000","last_undegraded":"2024-01-16T13:52:06.382533+0000","last_fullsized":"2024-01-16T13:52:06.382533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:05.397603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904528+0000","last_change":"2024-01-16T13:52:04.926700+0000","last_active":"2024-01-16T13:52:05.904528+0000","last_peered":"2024-01-16T13:52:05.904528+0000","last_clean":"2024-01-16T13:52:05.904528+0000","last_became_active":"2024-01-16T13:52:04.923728+0000","last_became_peered":"2024-01-16T13:52:04.923728+0000","last_unstale":"2024-01-16T13:52:05.904528+0000","last_undegraded":"2024-01-16T13:52:05.904528+0000","last_fullsized":"2024-01-16T13:52:05.904528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382807+0000","last_change":"2024-01-16T13:52:04.931697+0000","last_active":"2024-01-16T13:52:06.382807+0000","last_peered":"2024-01-16T13:52:06.382807+0000","last_clean":"2024-01-16T13:52:06.382807+0000","last_became_active":"2024-01-16T13:52:04.931415+0000","last_became_peered":"2024-01-16T13:52:04.931415+0000","last_unstale":"2024-01-16T13:52:06.382807+0000","last_undegraded":"2024-01-16T13:52:06.382807+0000","last_fullsized":"2024-01-16T13:52:06.382807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904991+0000","last_change":"2024-01-16T13:52:04.925527+0000","last_active":"2024-01-16T13:52:05.904991+0000","last_peered":"2024-01-16T13:52:05.904991+0000","last_clean":"2024-01-16T13:52:05.904991+0000","last_became_active":"2024-01-16T13:52:04.923026+0000","last_became_peered":"2024-01-16T13:52:04.923026+0000","last_unstale":"2024-01-16T13:52:05.904991+0000","last_undegraded":"2024-01-16T13:52:05.904991+0000","last_fullsized":"2024-01-16T13:52:05.904991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382610+0000","last_change":"2024-01-16T13:52:04.930095+0000","last_active":"2024-01-16T13:52:06.382610+0000","last_peered":"2024-01-16T13:52:06.382610+0000","last_clean":"2024-01-16T13:52:06.382610+0000","last_became_active":"2024-01-16T13:52:04.926632+0000","last_became_peered":"2024-01-16T13:52:04.926632+0000","last_unstale":"2024-01-16T13:52:06.382610+0000","last_undegraded":"2024-01-16T13:52:06.382610+0000","last_fullsized":"2024-01-16T13:52:06.382610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382842+0000","last_change":"2024-01-16T13:52:04.927129+0000","last_active":"2024-01-16T13:52:06.382842+0000","last_peered":"2024-01-16T13:52:06.382842+0000","last_clean":"2024-01-16T13:52:06.382842+0000","last_became_active":"2024-01-16T13:52:04.926541+0000","last_became_peered":"2024-01-16T13:52:04.926541+0000","last_unstale":"2024-01-16T13:52:06.382842+0000","last_undegraded":"2024-01-16T13:52:06.382842+0000","last_fullsized":"2024-01-16T13:52:06.382842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383203+0000","last_change":"2024-01-16T13:52:04.935989+0000","last_active":"2024-01-16T13:52:06.383203+0000","last_peered":"2024-01-16T13:52:06.383203+0000","last_clean":"2024-01-16T13:52:06.383203+0000","last_became_active":"2024-01-16T13:52:04.935750+0000","last_became_peered":"2024-01-16T13:52:04.935750+0000","last_unstale":"2024-01-16T13:52:06.383203+0000","last_undegraded":"2024-01-16T13:52:06.383203+0000","last_fullsized":"2024-01-16T13:52:06.383203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904269+0000","last_change":"2024-01-16T13:52:04.941805+0000","last_active":"2024-01-16T13:52:05.904269+0000","last_peered":"2024-01-16T13:52:05.904269+0000","last_clean":"2024-01-16T13:52:05.904269+0000","last_became_active":"2024-01-16T13:52:04.941552+0000","last_became_peered":"2024-01-16T13:52:04.941552+0000","last_unstale":"2024-01-16T13:52:05.904269+0000","last_undegraded":"2024-01-16T13:52:05.904269+0000","last_fullsized":"2024-01-16T13:52:05.904269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904181+0000","last_change":"2024-01-16T13:52:04.941357+0000","last_active":"2024-01-16T13:52:05.904181+0000","last_peered":"2024-01-16T13:52:05.904181+0000","last_clean":"2024-01-16T13:52:05.904181+0000","last_became_active":"2024-01-16T13:52:04.940946+0000","last_became_peered":"2024-01-16T13:52:04.940946+0000","last_unstale":"2024-01-16T13:52:05.904181+0000","last_undegraded":"2024-01-16T13:52:05.904181+0000","last_fullsized":"2024-01-16T13:52:05.904181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383260+0000","last_change":"2024-01-16T13:52:04.923980+0000","last_active":"2024-01-16T13:52:06.383260+0000","last_peered":"2024-01-16T13:52:06.383260+0000","last_clean":"2024-01-16T13:52:06.383260+0000","last_became_active":"2024-01-16T13:52:04.923398+0000","last_became_peered":"2024-01-16T13:52:04.923398+0000","last_unstale":"2024-01-16T13:52:06.383260+0000","last_undegraded":"2024-01-16T13:52:06.383260+0000","last_fullsized":"2024-01-16T13:52:06.383260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383179+0000","last_change":"2024-01-16T13:52:04.922440+0000","last_active":"2024-01-16T13:52:06.383179+0000","last_peered":"2024-01-16T13:52:06.383179+0000","last_clean":"2024-01-16T13:52:06.383179+0000","last_became_active":"2024-01-16T13:52:04.922170+0000","last_became_peered":"2024-01-16T13:52:04.922170+0000","last_unstale":"2024-01-16T13:52:06.383179+0000","last_undegraded":"2024-01-16T13:52:06.383179+0000","last_fullsized":"2024-01-16T13:52:06.383179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382935+0000","last_change":"2024-01-16T13:52:04.918126+0000","last_active":"2024-01-16T13:52:06.382935+0000","last_peered":"2024-01-16T13:52:06.382935+0000","last_clean":"2024-01-16T13:52:06.382935+0000","last_became_active":"2024-01-16T13:52:04.917683+0000","last_became_peered":"2024-01-16T13:52:04.917683+0000","last_unstale":"2024-01-16T13:52:06.382935+0000","last_undegraded":"2024-01-16T13:52:06.382935+0000","last_fullsized":"2024-01-16T13:52:06.382935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904662+0000","last_change":"2024-01-16T13:52:04.919145+0000","last_active":"2024-01-16T13:52:05.904662+0000","last_peered":"2024-01-16T13:52:05.904662+0000","last_clean":"2024-01-16T13:52:05.904662+0000","last_became_active":"2024-01-16T13:52:04.918897+0000","last_became_peered":"2024-01-16T13:52:04.918897+0000","last_unstale":"2024-01-16T13:52:05.904662+0000","last_undegraded":"2024-01-16T13:52:05.904662+0000","last_fullsized":"2024-01-16T13:52:05.904662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.923697+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.923053+0000","last_became_peered":"2024-01-16T13:52:04.923053+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904261+0000","last_change":"2024-01-16T13:52:04.927671+0000","last_active":"2024-01-16T13:52:05.904261+0000","last_peered":"2024-01-16T13:52:05.904261+0000","last_clean":"2024-01-16T13:52:05.904261+0000","last_became_active":"2024-01-16T13:52:04.927402+0000","last_became_peered":"2024-01-16T13:52:04.927402+0000","last_unstale":"2024-01-16T13:52:05.904261+0000","last_undegraded":"2024-01-16T13:52:05.904261+0000","last_fullsized":"2024-01-16T13:52:05.904261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382408+0000","last_change":"2024-01-16T13:52:04.924972+0000","last_active":"2024-01-16T13:52:06.382408+0000","last_peered":"2024-01-16T13:52:06.382408+0000","last_clean":"2024-01-16T13:52:06.382408+0000","last_became_active":"2024-01-16T13:52:04.924756+0000","last_became_peered":"2024-01-16T13:52:04.924756+0000","last_unstale":"2024-01-16T13:52:06.382408+0000","last_undegraded":"2024-01-16T13:52:06.382408+0000","last_fullsized":"2024-01-16T13:52:06.382408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383026+0000","last_change":"2024-01-16T13:52:04.932352+0000","last_active":"2024-01-16T13:52:06.383026+0000","last_peered":"2024-01-16T13:52:06.383026+0000","last_clean":"2024-01-16T13:52:06.383026+0000","last_became_active":"2024-01-16T13:52:04.926615+0000","last_became_peered":"2024-01-16T13:52:04.926615+0000","last_unstale":"2024-01-16T13:52:06.383026+0000","last_undegraded":"2024-01-16T13:52:06.383026+0000","last_fullsized":"2024-01-16T13:52:06.383026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382783+0000","last_change":"2024-01-16T13:52:03.906795+0000","last_active":"2024-01-16T13:52:06.382783+0000","last_peered":"2024-01-16T13:52:06.382783+0000","last_clean":"2024-01-16T13:52:06.382783+0000","last_became_active":"2024-01-16T13:52:03.906413+0000","last_became_peered":"2024-01-16T13:52:03.906413+0000","last_unstale":"2024-01-16T13:52:06.382783+0000","last_undegraded":"2024-01-16T13:52:06.382783+0000","last_fullsized":"2024-01-16T13:52:06.382783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383205+0000","last_change":"2024-01-16T13:52:04.922947+0000","last_active":"2024-01-16T13:52:06.383205+0000","last_peered":"2024-01-16T13:52:06.383205+0000","last_clean":"2024-01-16T13:52:06.383205+0000","last_became_active":"2024-01-16T13:52:04.922641+0000","last_became_peered":"2024-01-16T13:52:04.922641+0000","last_unstale":"2024-01-16T13:52:06.383205+0000","last_undegraded":"2024-01-16T13:52:06.383205+0000","last_fullsized":"2024-01-16T13:52:06.383205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383275+0000","last_change":"2024-01-16T13:52:03.911364+0000","last_active":"2024-01-16T13:52:06.383275+0000","last_peered":"2024-01-16T13:52:06.383275+0000","last_clean":"2024-01-16T13:52:06.383275+0000","last_became_active":"2024-01-16T13:52:03.911096+0000","last_became_peered":"2024-01-16T13:52:03.911096+0000","last_unstale":"2024-01-16T13:52:06.383275+0000","last_undegraded":"2024-01-16T13:52:06.383275+0000","last_fullsized":"2024-01-16T13:52:06.383275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382855+0000","last_change":"2024-01-16T13:52:04.941583+0000","last_active":"2024-01-16T13:52:06.382855+0000","last_peered":"2024-01-16T13:52:06.382855+0000","last_clean":"2024-01-16T13:52:06.382855+0000","last_became_active":"2024-01-16T13:52:04.941270+0000","last_became_peered":"2024-01-16T13:52:04.941270+0000","last_unstale":"2024-01-16T13:52:06.382855+0000","last_undegraded":"2024-01-16T13:52:06.382855+0000","last_fullsized":"2024-01-16T13:52:06.382855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383462+0000","last_change":"2024-01-16T13:52:03.908013+0000","last_active":"2024-01-16T13:52:06.383462+0000","last_peered":"2024-01-16T13:52:06.383462+0000","last_clean":"2024-01-16T13:52:06.383462+0000","last_became_active":"2024-01-16T13:52:03.907716+0000","last_became_peered":"2024-01-16T13:52:03.907716+0000","last_unstale":"2024-01-16T13:52:06.383462+0000","last_undegraded":"2024-01-16T13:52:06.383462+0000","last_fullsized":"2024-01-16T13:52:06.383462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904017+0000","last_change":"2024-01-16T13:52:04.920391+0000","last_active":"2024-01-16T13:52:05.904017+0000","last_peered":"2024-01-16T13:52:05.904017+0000","last_clean":"2024-01-16T13:52:05.904017+0000","last_became_active":"2024-01-16T13:52:04.920118+0000","last_became_peered":"2024-01-16T13:52:04.920118+0000","last_unstale":"2024-01-16T13:52:05.904017+0000","last_undegraded":"2024-01-16T13:52:05.904017+0000","last_fullsized":"2024-01-16T13:52:05.904017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382906+0000","last_change":"2024-01-16T13:52:03.905310+0000","last_active":"2024-01-16T13:52:06.382906+0000","last_peered":"2024-01-16T13:52:06.382906+0000","last_clean":"2024-01-16T13:52:06.382906+0000","last_became_active":"2024-01-16T13:52:03.905031+0000","last_became_peered":"2024-01-16T13:52:03.905031+0000","last_unstale":"2024-01-16T13:52:06.382906+0000","last_undegraded":"2024-01-16T13:52:06.382906+0000","last_fullsized":"2024-01-16T13:52:06.382906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904705+0000","last_change":"2024-01-16T13:52:04.912084+0000","last_active":"2024-01-16T13:52:05.904705+0000","last_peered":"2024-01-16T13:52:05.904705+0000","last_clean":"2024-01-16T13:52:05.904705+0000","last_became_active":"2024-01-16T13:52:04.911792+0000","last_became_peered":"2024-01-16T13:52:04.911792+0000","last_unstale":"2024-01-16T13:52:05.904705+0000","last_undegraded":"2024-01-16T13:52:05.904705+0000","last_fullsized":"2024-01-16T13:52:05.904705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383224+0000","last_change":"2024-01-16T13:52:03.912985+0000","last_active":"2024-01-16T13:52:06.383224+0000","last_peered":"2024-01-16T13:52:06.383224+0000","last_clean":"2024-01-16T13:52:06.383224+0000","last_became_active":"2024-01-16T13:52:03.912758+0000","last_became_peered":"2024-01-16T13:52:03.912758+0000","last_unstale":"2024-01-16T13:52:06.383224+0000","last_undegraded":"2024-01-16T13:52:06.383224+0000","last_fullsized":"2024-01-16T13:52:06.383224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382525+0000","last_change":"2024-01-16T13:52:04.917347+0000","last_active":"2024-01-16T13:52:06.382525+0000","last_peered":"2024-01-16T13:52:06.382525+0000","last_clean":"2024-01-16T13:52:06.382525+0000","last_became_active":"2024-01-16T13:52:04.916918+0000","last_became_peered":"2024-01-16T13:52:04.916918+0000","last_unstale":"2024-01-16T13:52:06.382525+0000","last_undegraded":"2024-01-16T13:52:06.382525+0000","last_fullsized":"2024-01-16T13:52:06.382525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:51.832998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903890+0000","last_change":"2024-01-16T13:52:03.903840+0000","last_active":"2024-01-16T13:52:05.903890+0000","last_peered":"2024-01-16T13:52:05.903890+0000","last_clean":"2024-01-16T13:52:05.903890+0000","last_became_active":"2024-01-16T13:52:03.903524+0000","last_became_peered":"2024-01-16T13:52:03.903524+0000","last_unstale":"2024-01-16T13:52:05.903890+0000","last_undegraded":"2024-01-16T13:52:05.903890+0000","last_fullsized":"2024-01-16T13:52:05.903890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383068+0000","last_change":"2024-01-16T13:52:04.924992+0000","last_active":"2024-01-16T13:52:06.383068+0000","last_peered":"2024-01-16T13:52:06.383068+0000","last_clean":"2024-01-16T13:52:06.383068+0000","last_became_active":"2024-01-16T13:52:04.924689+0000","last_became_peered":"2024-01-16T13:52:04.924689+0000","last_unstale":"2024-01-16T13:52:06.383068+0000","last_undegraded":"2024-01-16T13:52:06.383068+0000","last_fullsized":"2024-01-16T13:52:06.383068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383042+0000","last_change":"2024-01-16T13:52:03.912658+0000","last_active":"2024-01-16T13:52:06.383042+0000","last_peered":"2024-01-16T13:52:06.383042+0000","last_clean":"2024-01-16T13:52:06.383042+0000","last_became_active":"2024-01-16T13:52:03.912382+0000","last_became_peered":"2024-01-16T13:52:03.912382+0000","last_unstale":"2024-01-16T13:52:06.383042+0000","last_undegraded":"2024-01-16T13:52:06.383042+0000","last_fullsized":"2024-01-16T13:52:06.383042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383509+0000","last_change":"2024-01-16T13:52:04.920332+0000","last_active":"2024-01-16T13:52:06.383509+0000","last_peered":"2024-01-16T13:52:06.383509+0000","last_clean":"2024-01-16T13:52:06.383509+0000","last_became_active":"2024-01-16T13:52:04.920061+0000","last_became_peered":"2024-01-16T13:52:04.920061+0000","last_unstale":"2024-01-16T13:52:06.383509+0000","last_undegraded":"2024-01-16T13:52:06.383509+0000","last_fullsized":"2024-01-16T13:52:06.383509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904985+0000","last_change":"2024-01-16T13:52:03.897543+0000","last_active":"2024-01-16T13:52:05.904985+0000","last_peered":"2024-01-16T13:52:05.904985+0000","last_clean":"2024-01-16T13:52:05.904985+0000","last_became_active":"2024-01-16T13:52:03.897059+0000","last_became_peered":"2024-01-16T13:52:03.897059+0000","last_unstale":"2024-01-16T13:52:05.904985+0000","last_undegraded":"2024-01-16T13:52:05.904985+0000","last_fullsized":"2024-01-16T13:52:05.904985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904732+0000","last_change":"2024-01-16T13:52:04.906828+0000","last_active":"2024-01-16T13:52:05.904732+0000","last_peered":"2024-01-16T13:52:05.904732+0000","last_clean":"2024-01-16T13:52:05.904732+0000","last_became_active":"2024-01-16T13:52:04.906549+0000","last_became_peered":"2024-01-16T13:52:04.906549+0000","last_unstale":"2024-01-16T13:52:05.904732+0000","last_undegraded":"2024-01-16T13:52:05.904732+0000","last_fullsized":"2024-01-16T13:52:05.904732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904051+0000","last_change":"2024-01-16T13:52:03.901711+0000","last_active":"2024-01-16T13:52:05.904051+0000","last_peered":"2024-01-16T13:52:05.904051+0000","last_clean":"2024-01-16T13:52:05.904051+0000","last_became_active":"2024-01-16T13:52:03.901420+0000","last_became_peered":"2024-01-16T13:52:03.901420+0000","last_unstale":"2024-01-16T13:52:05.904051+0000","last_undegraded":"2024-01-16T13:52:05.904051+0000","last_fullsized":"2024-01-16T13:52:05.904051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904698+0000","last_change":"2024-01-16T13:52:04.935326+0000","last_active":"2024-01-16T13:52:05.904698+0000","last_peered":"2024-01-16T13:52:05.904698+0000","last_clean":"2024-01-16T13:52:05.904698+0000","last_became_active":"2024-01-16T13:52:04.934628+0000","last_became_peered":"2024-01-16T13:52:04.934628+0000","last_unstale":"2024-01-16T13:52:05.904698+0000","last_undegraded":"2024-01-16T13:52:05.904698+0000","last_fullsized":"2024-01-16T13:52:05.904698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904576+0000","last_change":"2024-01-16T13:52:03.902616+0000","last_active":"2024-01-16T13:52:05.904576+0000","last_peered":"2024-01-16T13:52:05.904576+0000","last_clean":"2024-01-16T13:52:05.904576+0000","last_became_active":"2024-01-16T13:52:03.902333+0000","last_became_peered":"2024-01-16T13:52:03.902333+0000","last_unstale":"2024-01-16T13:52:05.904576+0000","last_undegraded":"2024-01-16T13:52:05.904576+0000","last_fullsized":"2024-01-16T13:52:05.904576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383250+0000","last_change":"2024-01-16T13:52:04.936172+0000","last_active":"2024-01-16T13:52:06.383250+0000","last_peered":"2024-01-16T13:52:06.383250+0000","last_clean":"2024-01-16T13:52:06.383250+0000","last_became_active":"2024-01-16T13:52:04.935881+0000","last_became_peered":"2024-01-16T13:52:04.935881+0000","last_unstale":"2024-01-16T13:52:06.383250+0000","last_undegraded":"2024-01-16T13:52:06.383250+0000","last_fullsized":"2024-01-16T13:52:06.383250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904808+0000","last_change":"2024-01-16T13:52:03.896280+0000","last_active":"2024-01-16T13:52:05.904808+0000","last_peered":"2024-01-16T13:52:05.904808+0000","last_clean":"2024-01-16T13:52:05.904808+0000","last_became_active":"2024-01-16T13:52:03.896001+0000","last_became_peered":"2024-01-16T13:52:03.896001+0000","last_unstale":"2024-01-16T13:52:05.904808+0000","last_undegraded":"2024-01-16T13:52:05.904808+0000","last_fullsized":"2024-01-16T13:52:05.904808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383116+0000","last_change":"2024-01-16T13:52:04.927368+0000","last_active":"2024-01-16T13:52:06.383116+0000","last_peered":"2024-01-16T13:52:06.383116+0000","last_clean":"2024-01-16T13:52:06.383116+0000","last_became_active":"2024-01-16T13:52:04.927100+0000","last_became_peered":"2024-01-16T13:52:04.927100+0000","last_unstale":"2024-01-16T13:52:06.383116+0000","last_undegraded":"2024-01-16T13:52:06.383116+0000","last_fullsized":"2024-01-16T13:52:06.383116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904369+0000","last_change":"2024-01-16T13:52:03.910360+0000","last_active":"2024-01-16T13:52:05.904369+0000","last_peered":"2024-01-16T13:52:05.904369+0000","last_clean":"2024-01-16T13:52:05.904369+0000","last_became_active":"2024-01-16T13:52:03.910098+0000","last_became_peered":"2024-01-16T13:52:03.910098+0000","last_unstale":"2024-01-16T13:52:05.904369+0000","last_undegraded":"2024-01-16T13:52:05.904369+0000","last_fullsized":"2024-01-16T13:52:05.904369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383195+0000","last_change":"2024-01-16T13:52:04.932597+0000","last_active":"2024-01-16T13:52:06.383195+0000","last_peered":"2024-01-16T13:52:06.383195+0000","last_clean":"2024-01-16T13:52:06.383195+0000","last_became_active":"2024-01-16T13:52:04.930511+0000","last_became_peered":"2024-01-16T13:52:04.930511+0000","last_unstale":"2024-01-16T13:52:06.383195+0000","last_undegraded":"2024-01-16T13:52:06.383195+0000","last_fullsized":"2024-01-16T13:52:06.383195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382997+0000","last_change":"2024-01-16T13:52:03.906336+0000","last_active":"2024-01-16T13:52:06.382997+0000","last_peered":"2024-01-16T13:52:06.382997+0000","last_clean":"2024-01-16T13:52:06.382997+0000","last_became_active":"2024-01-16T13:52:03.905656+0000","last_became_peered":"2024-01-16T13:52:03.905656+0000","last_unstale":"2024-01-16T13:52:06.382997+0000","last_undegraded":"2024-01-16T13:52:06.382997+0000","last_fullsized":"2024-01-16T13:52:06.382997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382700+0000","last_change":"2024-01-16T13:52:04.927208+0000","last_active":"2024-01-16T13:52:06.382700+0000","last_peered":"2024-01-16T13:52:06.382700+0000","last_clean":"2024-01-16T13:52:06.382700+0000","last_became_active":"2024-01-16T13:52:04.926931+0000","last_became_peered":"2024-01-16T13:52:04.926931+0000","last_unstale":"2024-01-16T13:52:06.382700+0000","last_undegraded":"2024-01-16T13:52:06.382700+0000","last_fullsized":"2024-01-16T13:52:06.382700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905228+0000","last_change":"2024-01-16T13:52:03.911290+0000","last_active":"2024-01-16T13:52:05.905228+0000","last_peered":"2024-01-16T13:52:05.905228+0000","last_clean":"2024-01-16T13:52:05.905228+0000","last_became_active":"2024-01-16T13:52:03.911029+0000","last_became_peered":"2024-01-16T13:52:03.911029+0000","last_unstale":"2024-01-16T13:52:05.905228+0000","last_undegraded":"2024-01-16T13:52:05.905228+0000","last_fullsized":"2024-01-16T13:52:05.905228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:40.789717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383244+0000","last_change":"2024-01-16T13:52:04.932349+0000","last_active":"2024-01-16T13:52:06.383244+0000","last_peered":"2024-01-16T13:52:06.383244+0000","last_clean":"2024-01-16T13:52:06.383244+0000","last_became_active":"2024-01-16T13:52:04.931854+0000","last_became_peered":"2024-01-16T13:52:04.931854+0000","last_unstale":"2024-01-16T13:52:06.383244+0000","last_undegraded":"2024-01-16T13:52:06.383244+0000","last_fullsized":"2024-01-16T13:52:06.383244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903653+0000","last_change":"2024-01-16T13:52:03.903815+0000","last_active":"2024-01-16T13:52:05.903653+0000","last_peered":"2024-01-16T13:52:05.903653+0000","last_clean":"2024-01-16T13:52:05.903653+0000","last_became_active":"2024-01-16T13:52:03.903487+0000","last_became_peered":"2024-01-16T13:52:03.903487+0000","last_unstale":"2024-01-16T13:52:05.903653+0000","last_undegraded":"2024-01-16T13:52:05.903653+0000","last_fullsized":"2024-01-16T13:52:05.903653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:49.732984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903921+0000","last_change":"2024-01-16T13:52:04.922443+0000","last_active":"2024-01-16T13:52:05.903921+0000","last_peered":"2024-01-16T13:52:05.903921+0000","last_clean":"2024-01-16T13:52:05.903921+0000","last_became_active":"2024-01-16T13:52:04.922131+0000","last_became_peered":"2024-01-16T13:52:04.922131+0000","last_unstale":"2024-01-16T13:52:05.903921+0000","last_undegraded":"2024-01-16T13:52:05.903921+0000","last_fullsized":"2024-01-16T13:52:05.903921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382223+0000","last_change":"2024-01-16T13:52:03.910304+0000","last_active":"2024-01-16T13:52:06.382223+0000","last_peered":"2024-01-16T13:52:06.382223+0000","last_clean":"2024-01-16T13:52:06.382223+0000","last_became_active":"2024-01-16T13:52:03.910009+0000","last_became_peered":"2024-01-16T13:52:03.910009+0000","last_unstale":"2024-01-16T13:52:06.382223+0000","last_undegraded":"2024-01-16T13:52:06.382223+0000","last_fullsized":"2024-01-16T13:52:06.382223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382979+0000","last_change":"2024-01-16T13:52:04.930880+0000","last_active":"2024-01-16T13:52:06.382979+0000","last_peered":"2024-01-16T13:52:06.382979+0000","last_clean":"2024-01-16T13:52:06.382979+0000","last_became_active":"2024-01-16T13:52:04.930553+0000","last_became_peered":"2024-01-16T13:52:04.930553+0000","last_unstale":"2024-01-16T13:52:06.382979+0000","last_undegraded":"2024-01-16T13:52:06.382979+0000","last_fullsized":"2024-01-16T13:52:06.382979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382946+0000","last_change":"2024-01-16T13:52:03.906573+0000","last_active":"2024-01-16T13:52:06.382946+0000","last_peered":"2024-01-16T13:52:06.382946+0000","last_clean":"2024-01-16T13:52:06.382946+0000","last_became_active":"2024-01-16T13:52:03.906257+0000","last_became_peered":"2024-01-16T13:52:03.906257+0000","last_unstale":"2024-01-16T13:52:06.382946+0000","last_undegraded":"2024-01-16T13:52:06.382946+0000","last_fullsized":"2024-01-16T13:52:06.382946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:14:03.506561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382905+0000","last_change":"2024-01-16T13:52:04.931581+0000","last_active":"2024-01-16T13:52:06.382905+0000","last_peered":"2024-01-16T13:52:06.382905+0000","last_clean":"2024-01-16T13:52:06.382905+0000","last_became_active":"2024-01-16T13:52:04.931306+0000","last_became_peered":"2024-01-16T13:52:04.931306+0000","last_unstale":"2024-01-16T13:52:06.382905+0000","last_undegraded":"2024-01-16T13:52:06.382905+0000","last_fullsized":"2024-01-16T13:52:06.382905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904968+0000","last_change":"2024-01-16T13:52:03.903873+0000","last_active":"2024-01-16T13:52:05.904968+0000","last_peered":"2024-01-16T13:52:05.904968+0000","last_clean":"2024-01-16T13:52:05.904968+0000","last_became_active":"2024-01-16T13:52:03.903582+0000","last_became_peered":"2024-01-16T13:52:03.903582+0000","last_unstale":"2024-01-16T13:52:05.904968+0000","last_undegraded":"2024-01-16T13:52:05.904968+0000","last_fullsized":"2024-01-16T13:52:05.904968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904494+0000","last_change":"2024-01-16T13:52:04.926087+0000","last_active":"2024-01-16T13:52:05.904494+0000","last_peered":"2024-01-16T13:52:05.904494+0000","last_clean":"2024-01-16T13:52:05.904494+0000","last_became_active":"2024-01-16T13:52:04.925736+0000","last_became_peered":"2024-01-16T13:52:04.925736+0000","last_unstale":"2024-01-16T13:52:05.904494+0000","last_undegraded":"2024-01-16T13:52:05.904494+0000","last_fullsized":"2024-01-16T13:52:05.904494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905152+0000","last_change":"2024-01-16T13:52:03.901400+0000","last_active":"2024-01-16T13:52:05.905152+0000","last_peered":"2024-01-16T13:52:05.905152+0000","last_clean":"2024-01-16T13:52:05.905152+0000","last_became_active":"2024-01-16T13:52:03.901087+0000","last_became_peered":"2024-01-16T13:52:03.901087+0000","last_unstale":"2024-01-16T13:52:05.905152+0000","last_undegraded":"2024-01-16T13:52:05.905152+0000","last_fullsized":"2024-01-16T13:52:05.905152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:43.211649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382982+0000","last_change":"2024-01-16T13:52:04.917916+0000","last_active":"2024-01-16T13:52:06.382982+0000","last_peered":"2024-01-16T13:52:06.382982+0000","last_clean":"2024-01-16T13:52:06.382982+0000","last_became_active":"2024-01-16T13:52:04.917649+0000","last_became_peered":"2024-01-16T13:52:04.917649+0000","last_unstale":"2024-01-16T13:52:06.382982+0000","last_undegraded":"2024-01-16T13:52:06.382982+0000","last_fullsized":"2024-01-16T13:52:06.382982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:37.904615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904621+0000","last_change":"2024-01-16T13:52:04.916017+0000","last_active":"2024-01-16T13:52:05.904621+0000","last_peered":"2024-01-16T13:52:05.904621+0000","last_clean":"2024-01-16T13:52:05.904621+0000","last_became_active":"2024-01-16T13:52:04.915765+0000","last_became_peered":"2024-01-16T13:52:04.915765+0000","last_unstale":"2024-01-16T13:52:05.904621+0000","last_undegraded":"2024-01-16T13:52:05.904621+0000","last_fullsized":"2024-01-16T13:52:05.904621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382686+0000","last_change":"2024-01-16T13:52:03.910443+0000","last_active":"2024-01-16T13:52:06.382686+0000","last_peered":"2024-01-16T13:52:06.382686+0000","last_clean":"2024-01-16T13:52:06.382686+0000","last_became_active":"2024-01-16T13:52:03.910025+0000","last_became_peered":"2024-01-16T13:52:03.910025+0000","last_unstale":"2024-01-16T13:52:06.382686+0000","last_undegraded":"2024-01-16T13:52:06.382686+0000","last_fullsized":"2024-01-16T13:52:06.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:22.047669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383379+0000","last_change":"2024-01-16T13:52:04.927591+0000","last_active":"2024-01-16T13:52:06.383379+0000","last_peered":"2024-01-16T13:52:06.383379+0000","last_clean":"2024-01-16T13:52:06.383379+0000","last_became_active":"2024-01-16T13:52:04.927020+0000","last_became_peered":"2024-01-16T13:52:04.927020+0000","last_unstale":"2024-01-16T13:52:06.383379+0000","last_undegraded":"2024-01-16T13:52:06.383379+0000","last_fullsized":"2024-01-16T13:52:06.383379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903799+0000","last_change":"2024-01-16T13:52:03.906447+0000","last_active":"2024-01-16T13:52:05.903799+0000","last_peered":"2024-01-16T13:52:05.903799+0000","last_clean":"2024-01-16T13:52:05.903799+0000","last_became_active":"2024-01-16T13:52:03.906082+0000","last_became_peered":"2024-01-16T13:52:03.906082+0000","last_unstale":"2024-01-16T13:52:05.903799+0000","last_undegraded":"2024-01-16T13:52:05.903799+0000","last_fullsized":"2024-01-16T13:52:05.903799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T13:52:04.913338+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:04.913338+0000","last_peered":"2024-01-16T13:52:04.913338+0000","last_clean":"2024-01-16T13:52:04.913338+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:04.913338+0000","last_undegraded":"2024-01-16T13:52:04.913338+0000","last_fullsized":"2024-01-16T13:52:04.913338+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382590+0000","last_change":"2024-01-16T13:52:04.923544+0000","last_active":"2024-01-16T13:52:06.382590+0000","last_peered":"2024-01-16T13:52:06.382590+0000","last_clean":"2024-01-16T13:52:06.382590+0000","last_became_active":"2024-01-16T13:52:04.923283+0000","last_became_peered":"2024-01-16T13:52:04.923283+0000","last_unstale":"2024-01-16T13:52:06.382590+0000","last_undegraded":"2024-01-16T13:52:06.382590+0000","last_fullsized":"2024-01-16T13:52:06.382590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382524+0000","last_change":"2024-01-16T13:52:03.910495+0000","last_active":"2024-01-16T13:52:06.382524+0000","last_peered":"2024-01-16T13:52:06.382524+0000","last_clean":"2024-01-16T13:52:06.382524+0000","last_became_active":"2024-01-16T13:52:03.910162+0000","last_became_peered":"2024-01-16T13:52:03.910162+0000","last_unstale":"2024-01-16T13:52:06.382524+0000","last_undegraded":"2024-01-16T13:52:06.382524+0000","last_fullsized":"2024-01-16T13:52:06.382524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:44.922338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:03.910098+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:03.909843+0000","last_became_peered":"2024-01-16T13:52:03.909843+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904899+0000","last_change":"2024-01-16T13:52:04.915509+0000","last_active":"2024-01-16T13:52:05.904899+0000","last_peered":"2024-01-16T13:52:05.904899+0000","last_clean":"2024-01-16T13:52:05.904899+0000","last_became_active":"2024-01-16T13:52:04.915240+0000","last_became_peered":"2024-01-16T13:52:04.915240+0000","last_unstale":"2024-01-16T13:52:05.904899+0000","last_undegraded":"2024-01-16T13:52:05.904899+0000","last_fullsized":"2024-01-16T13:52:05.904899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:07.544081+0000","last_change":"2024-01-16T13:52:03.903483+0000","last_active":"2024-01-16T13:52:07.544081+0000","last_peered":"2024-01-16T13:52:07.544081+0000","last_clean":"2024-01-16T13:52:07.544081+0000","last_became_active":"2024-01-16T13:52:03.903147+0000","last_became_peered":"2024-01-16T13:52:03.903147+0000","last_unstale":"2024-01-16T13:52:07.544081+0000","last_undegraded":"2024-01-16T13:52:07.544081+0000","last_fullsized":"2024-01-16T13:52:07.544081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383351+0000","last_change":"2024-01-16T13:52:04.927887+0000","last_active":"2024-01-16T13:52:06.383351+0000","last_peered":"2024-01-16T13:52:06.383351+0000","last_clean":"2024-01-16T13:52:06.383351+0000","last_became_active":"2024-01-16T13:52:04.927626+0000","last_became_peered":"2024-01-16T13:52:04.927626+0000","last_unstale":"2024-01-16T13:52:06.383351+0000","last_undegraded":"2024-01-16T13:52:06.383351+0000","last_fullsized":"2024-01-16T13:52:06.383351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904612+0000","last_change":"2024-01-16T13:52:03.903609+0000","last_active":"2024-01-16T13:52:05.904612+0000","last_peered":"2024-01-16T13:52:05.904612+0000","last_clean":"2024-01-16T13:52:05.904612+0000","last_became_active":"2024-01-16T13:52:03.903258+0000","last_became_peered":"2024-01-16T13:52:03.903258+0000","last_unstale":"2024-01-16T13:52:05.904612+0000","last_undegraded":"2024-01-16T13:52:05.904612+0000","last_fullsized":"2024-01-16T13:52:05.904612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383138+0000","last_change":"2024-01-16T13:52:04.918025+0000","last_active":"2024-01-16T13:52:06.383138+0000","last_peered":"2024-01-16T13:52:06.383138+0000","last_clean":"2024-01-16T13:52:06.383138+0000","last_became_active":"2024-01-16T13:52:04.917716+0000","last_became_peered":"2024-01-16T13:52:04.917716+0000","last_unstale":"2024-01-16T13:52:06.383138+0000","last_undegraded":"2024-01-16T13:52:06.383138+0000","last_fullsized":"2024-01-16T13:52:06.383138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382341+0000","last_change":"2024-01-16T13:52:03.909845+0000","last_active":"2024-01-16T13:52:06.382341+0000","last_peered":"2024-01-16T13:52:06.382341+0000","last_clean":"2024-01-16T13:52:06.382341+0000","last_became_active":"2024-01-16T13:52:03.909573+0000","last_became_peered":"2024-01-16T13:52:03.909573+0000","last_unstale":"2024-01-16T13:52:06.382341+0000","last_undegraded":"2024-01-16T13:52:06.382341+0000","last_fullsized":"2024-01-16T13:52:06.382341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:04.918497+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:04.918198+0000","last_became_peered":"2024-01-16T13:52:04.918198+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:01.775696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904593+0000","last_change":"2024-01-16T13:52:03.904013+0000","last_active":"2024-01-16T13:52:05.904593+0000","last_peered":"2024-01-16T13:52:05.904593+0000","last_clean":"2024-01-16T13:52:05.904593+0000","last_became_active":"2024-01-16T13:52:03.903827+0000","last_became_peered":"2024-01-16T13:52:03.903827+0000","last_unstale":"2024-01-16T13:52:05.904593+0000","last_undegraded":"2024-01-16T13:52:05.904593+0000","last_fullsized":"2024-01-16T13:52:05.904593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383603+0000","last_change":"2024-01-16T13:52:04.912859+0000","last_active":"2024-01-16T13:52:06.383603+0000","last_peered":"2024-01-16T13:52:06.383603+0000","last_clean":"2024-01-16T13:52:06.383603+0000","last_became_active":"2024-01-16T13:52:04.912597+0000","last_became_peered":"2024-01-16T13:52:04.912597+0000","last_unstale":"2024-01-16T13:52:06.383603+0000","last_undegraded":"2024-01-16T13:52:06.383603+0000","last_fullsized":"2024-01-16T13:52:06.383603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:03.908118+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:03.907800+0000","last_became_peered":"2024-01-16T13:52:03.907800+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383308+0000","last_change":"2024-01-16T13:52:04.918302+0000","last_active":"2024-01-16T13:52:06.383308+0000","last_peered":"2024-01-16T13:52:06.383308+0000","last_clean":"2024-01-16T13:52:06.383308+0000","last_became_active":"2024-01-16T13:52:04.917994+0000","last_became_peered":"2024-01-16T13:52:04.917994+0000","last_unstale":"2024-01-16T13:52:06.383308+0000","last_undegraded":"2024-01-16T13:52:06.383308+0000","last_fullsized":"2024-01-16T13:52:06.383308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383067+0000","last_change":"2024-01-16T13:52:03.903338+0000","last_active":"2024-01-16T13:52:06.383067+0000","last_peered":"2024-01-16T13:52:06.383067+0000","last_clean":"2024-01-16T13:52:06.383067+0000","last_became_active":"2024-01-16T13:52:03.903053+0000","last_became_peered":"2024-01-16T13:52:03.903053+0000","last_unstale":"2024-01-16T13:52:06.383067+0000","last_undegraded":"2024-01-16T13:52:06.383067+0000","last_fullsized":"2024-01-16T13:52:06.383067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:27.027544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903790+0000","last_change":"2024-01-16T13:52:04.924256+0000","last_active":"2024-01-16T13:52:05.903790+0000","last_peered":"2024-01-16T13:52:05.903790+0000","last_clean":"2024-01-16T13:52:05.903790+0000","last_became_active":"2024-01-16T13:52:04.923945+0000","last_became_peered":"2024-01-16T13:52:04.923945+0000","last_unstale":"2024-01-16T13:52:05.903790+0000","last_undegraded":"2024-01-16T13:52:05.903790+0000","last_fullsized":"2024-01-16T13:52:05.903790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382587+0000","last_change":"2024-01-16T13:52:03.906252+0000","last_active":"2024-01-16T13:52:06.382587+0000","last_peered":"2024-01-16T13:52:06.382587+0000","last_clean":"2024-01-16T13:52:06.382587+0000","last_became_active":"2024-01-16T13:52:03.905949+0000","last_became_peered":"2024-01-16T13:52:03.905949+0000","last_unstale":"2024-01-16T13:52:06.382587+0000","last_undegraded":"2024-01-16T13:52:06.382587+0000","last_fullsized":"2024-01-16T13:52:06.382587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:00.771273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:04.933410+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:04.932914+0000","last_became_peered":"2024-01-16T13:52:04.932914+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904753+0000","last_change":"2024-01-16T13:52:03.897651+0000","last_active":"2024-01-16T13:52:05.904753+0000","last_peered":"2024-01-16T13:52:05.904753+0000","last_clean":"2024-01-16T13:52:05.904753+0000","last_became_active":"2024-01-16T13:52:03.897380+0000","last_became_peered":"2024-01-16T13:52:03.897380+0000","last_unstale":"2024-01-16T13:52:05.904753+0000","last_undegraded":"2024-01-16T13:52:05.904753+0000","last_fullsized":"2024-01-16T13:52:05.904753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904179+0000","last_change":"2024-01-16T13:52:04.934873+0000","last_active":"2024-01-16T13:52:05.904179+0000","last_peered":"2024-01-16T13:52:05.904179+0000","last_clean":"2024-01-16T13:52:05.904179+0000","last_became_active":"2024-01-16T13:52:04.934573+0000","last_became_peered":"2024-01-16T13:52:04.934573+0000","last_unstale":"2024-01-16T13:52:05.904179+0000","last_undegraded":"2024-01-16T13:52:05.904179+0000","last_fullsized":"2024-01-16T13:52:05.904179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:03.911272+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:03.910949+0000","last_became_peered":"2024-01-16T13:52:03.910949+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383514+0000","last_change":"2024-01-16T13:52:04.933704+0000","last_active":"2024-01-16T13:52:06.383514+0000","last_peered":"2024-01-16T13:52:06.383514+0000","last_clean":"2024-01-16T13:52:06.383514+0000","last_became_active":"2024-01-16T13:52:04.933390+0000","last_became_peered":"2024-01-16T13:52:04.933390+0000","last_unstale":"2024-01-16T13:52:06.383514+0000","last_undegraded":"2024-01-16T13:52:06.383514+0000","last_fullsized":"2024-01-16T13:52:06.383514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903638+0000","last_change":"2024-01-16T13:52:04.922828+0000","last_active":"2024-01-16T13:52:05.903638+0000","last_peered":"2024-01-16T13:52:05.903638+0000","last_clean":"2024-01-16T13:52:05.903638+0000","last_became_active":"2024-01-16T13:52:04.922428+0000","last_became_peered":"2024-01-16T13:52:04.922428+0000","last_unstale":"2024-01-16T13:52:05.903638+0000","last_undegraded":"2024-01-16T13:52:05.903638+0000","last_fullsized":"2024-01-16T13:52:05.903638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382275+0000","last_change":"2024-01-16T13:52:04.915322+0000","last_active":"2024-01-16T13:52:06.382275+0000","last_peered":"2024-01-16T13:52:06.382275+0000","last_clean":"2024-01-16T13:52:06.382275+0000","last_became_active":"2024-01-16T13:52:04.915031+0000","last_became_peered":"2024-01-16T13:52:04.915031+0000","last_unstale":"2024-01-16T13:52:06.382275+0000","last_undegraded":"2024-01-16T13:52:06.382275+0000","last_fullsized":"2024-01-16T13:52:06.382275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905142+0000","last_change":"2024-01-16T13:52:04.925541+0000","last_active":"2024-01-16T13:52:05.905142+0000","last_peered":"2024-01-16T13:52:05.905142+0000","last_clean":"2024-01-16T13:52:05.905142+0000","last_became_active":"2024-01-16T13:52:04.923411+0000","last_became_peered":"2024-01-16T13:52:04.923411+0000","last_unstale":"2024-01-16T13:52:05.905142+0000","last_undegraded":"2024-01-16T13:52:05.905142+0000","last_fullsized":"2024-01-16T13:52:05.905142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382903+0000","last_change":"2024-01-16T13:52:04.919996+0000","last_active":"2024-01-16T13:52:06.382903+0000","last_peered":"2024-01-16T13:52:06.382903+0000","last_clean":"2024-01-16T13:52:06.382903+0000","last_became_active":"2024-01-16T13:52:04.919257+0000","last_became_peered":"2024-01-16T13:52:04.919257+0000","last_unstale":"2024-01-16T13:52:06.382903+0000","last_undegraded":"2024-01-16T13:52:06.382903+0000","last_fullsized":"2024-01-16T13:52:06.382903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382991+0000","last_change":"2024-01-16T13:52:04.921215+0000","last_active":"2024-01-16T13:52:06.382991+0000","last_peered":"2024-01-16T13:52:06.382991+0000","last_clean":"2024-01-16T13:52:06.382991+0000","last_became_active":"2024-01-16T13:52:04.920992+0000","last_became_peered":"2024-01-16T13:52:04.920992+0000","last_unstale":"2024-01-16T13:52:06.382991+0000","last_undegraded":"2024-01-16T13:52:06.382991+0000","last_fullsized":"2024-01-16T13:52:06.382991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:04.911541+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:04.911263+0000","last_became_peered":"2024-01-16T13:52:04.911263+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904058+0000","last_change":"2024-01-16T13:52:04.932056+0000","last_active":"2024-01-16T13:52:05.904058+0000","last_peered":"2024-01-16T13:52:05.904058+0000","last_clean":"2024-01-16T13:52:05.904058+0000","last_became_active":"2024-01-16T13:52:04.931794+0000","last_became_peered":"2024-01-16T13:52:04.931794+0000","last_unstale":"2024-01-16T13:52:05.904058+0000","last_undegraded":"2024-01-16T13:52:05.904058+0000","last_fullsized":"2024-01-16T13:52:05.904058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383217+0000","last_change":"2024-01-16T13:52:04.918735+0000","last_active":"2024-01-16T13:52:06.383217+0000","last_peered":"2024-01-16T13:52:06.383217+0000","last_clean":"2024-01-16T13:52:06.383217+0000","last_became_active":"2024-01-16T13:52:04.918440+0000","last_became_peered":"2024-01-16T13:52:04.918440+0000","last_unstale":"2024-01-16T13:52:06.383217+0000","last_undegraded":"2024-01-16T13:52:06.383217+0000","last_fullsized":"2024-01-16T13:52:06.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905101+0000","last_change":"2024-01-16T13:52:04.923904+0000","last_active":"2024-01-16T13:52:05.905101+0000","last_peered":"2024-01-16T13:52:05.905101+0000","last_clean":"2024-01-16T13:52:05.905101+0000","last_became_active":"2024-01-16T13:52:04.923426+0000","last_became_peered":"2024-01-16T13:52:04.923426+0000","last_unstale":"2024-01-16T13:52:05.905101+0000","last_undegraded":"2024-01-16T13:52:05.905101+0000","last_fullsized":"2024-01-16T13:52:05.905101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383063+0000","last_change":"2024-01-16T13:52:04.919553+0000","last_active":"2024-01-16T13:52:06.383063+0000","last_peered":"2024-01-16T13:52:06.383063+0000","last_clean":"2024-01-16T13:52:06.383063+0000","last_became_active":"2024-01-16T13:52:04.919053+0000","last_became_peered":"2024-01-16T13:52:04.919053+0000","last_unstale":"2024-01-16T13:52:06.383063+0000","last_undegraded":"2024-01-16T13:52:06.383063+0000","last_fullsized":"2024-01-16T13:52:06.383063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905086+0000","last_change":"2024-01-16T13:52:04.926131+0000","last_active":"2024-01-16T13:52:05.905086+0000","last_peered":"2024-01-16T13:52:05.905086+0000","last_clean":"2024-01-16T13:52:05.905086+0000","last_became_active":"2024-01-16T13:52:04.925848+0000","last_became_peered":"2024-01-16T13:52:04.925848+0000","last_unstale":"2024-01-16T13:52:05.905086+0000","last_undegraded":"2024-01-16T13:52:05.905086+0000","last_fullsized":"2024-01-16T13:52:05.905086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382745+0000","last_change":"2024-01-16T13:52:04.931968+0000","last_active":"2024-01-16T13:52:06.382745+0000","last_peered":"2024-01-16T13:52:06.382745+0000","last_clean":"2024-01-16T13:52:06.382745+0000","last_became_active":"2024-01-16T13:52:04.931678+0000","last_became_peered":"2024-01-16T13:52:04.931678+0000","last_unstale":"2024-01-16T13:52:06.382745+0000","last_undegraded":"2024-01-16T13:52:06.382745+0000","last_fullsized":"2024-01-16T13:52:06.382745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904558+0000","last_change":"2024-01-16T13:52:04.929411+0000","last_active":"2024-01-16T13:52:05.904558+0000","last_peered":"2024-01-16T13:52:05.904558+0000","last_clean":"2024-01-16T13:52:05.904558+0000","last_became_active":"2024-01-16T13:52:04.929143+0000","last_became_peered":"2024-01-16T13:52:04.929143+0000","last_unstale":"2024-01-16T13:52:05.904558+0000","last_undegraded":"2024-01-16T13:52:05.904558+0000","last_fullsized":"2024-01-16T13:52:05.904558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:52:13.645 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:52:13.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.821+0000 7f6bc15d1700 1 Processor -- start 2024-01-16T13:52:13.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.821+0000 7f6bc15d1700 1 -- start start 2024-01-16T13:52:13.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.821+0000 7f6bc15d1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 0x7f6bbc168e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.821+0000 7f6bc15d1700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc0d2810 con 0x7f6bbc15e480 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.821+0000 7f6bc15d1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc0d2a70 con 0x7f6bbc0d3610 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.821+0000 7f6bc15d1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0d2d50 con 0x7f6bbc0b6060 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bbaffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6bbc15e480 0x7f6bbc0d2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39306/0 (socket says 172.21.15.113:39306) 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bbaffd700 1 -- 172.21.15.113:0/355231071 learned_addr learned my addr 172.21.15.113:0/355231071 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bbb7fe700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 0x7f6bbc168e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282506262 0 0) 0x7f6bbc0d2810 con 0x7f6bbc15e480 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba40035b0 con 0x7f6bbc15e480 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887271989 0 0) 0x7f6ba40035b0 con 0x7f6bbc15e480 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 msgr2=0x7f6bbc168e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 0x7f6bbc168e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 >> v1:172.21.15.124:6789/0 conn(0x7f6bbc0d3610 legacy=0x7f6bbc0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0d3030 con 0x7f6bbc15e480 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f6bb0001670 con 0x7f6bbc15e480 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 >> v1:172.21.15.113:6789/0 conn(0x7f6bbc15e480 legacy=0x7f6bbc0d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 0x7f6ba4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6ba4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ba4004d90 0x7f6ba4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0d2d50 con 0x7f6ba4003fb0 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba400bba0 con 0x7f6ba4004d90 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba400bd60 con 0x7f6bbc0b6060 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6ba4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bbaffd700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 0x7f6ba4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bbb7fe700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ba4004d90 0x7f6ba4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bba7fc700 1 -- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 msgr2=0x7f6ba4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc0b6060 0x7f6ba4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bba7fc700 1 -- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ba4004d90 msgr2=0x7f6ba4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ba4004d90 0x7f6ba4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bba7fc700 1 -- 172.21.15.113:0/355231071 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba40035b0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.822+0000 7f6bbb7fe700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ba4004d90 0x7f6ba4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6ba4004850 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6bac0017c0 tx=0x7f6ba4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5306999c60859d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac0052d0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac005430 con 0x7f6ba4003fb0 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bb9ffb700 1 -- 172.21.15.113:0/355231071 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac0038c0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bc15d1700 1 -- 172.21.15.113:0/355231071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 msgr2=0x7f6ba4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/355231071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6ba4004850 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6bac0017c0 tx=0x7f6ba4005880 comp rx=0 tx=0).stop 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bbb7fe700 1 -- 172.21.15.113:0/355231071 reap_dead start 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bc15d1700 1 -- 172.21.15.113:0/355231071 shutdown_connections 2024-01-16T13:52:13.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bc15d1700 1 -- 172.21.15.113:0/355231071 >> 172.21.15.113:0/355231071 conn(0x7f6bbc0d0950 msgr2=0x7f6bbc0d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:13.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bc15d1700 1 -- 172.21.15.113:0/355231071 shutdown_connections 2024-01-16T13:52:13.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.823+0000 7f6bc15d1700 1 -- 172.21.15.113:0/355231071 wait complete. 2024-01-16T13:52:13.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 Processor -- start 2024-01-16T13:52:13.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 -- start start 2024-01-16T13:52:13.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6bbc0c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bbc0cdb70 0x7f6bbc06b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc06b790 0x7f6bbc0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc06bc30 con 0x7f6ba4003fb0 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0c8730 con 0x7f6bbc0cdb70 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.825+0000 7f6bc15d1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0c8a10 con 0x7f6bbc06b790 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6bbc0c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6bbc0c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48244/0 (socket says 172.21.15.113:48244) 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 -- 172.21.15.113:0/693658250 learned_addr learned my addr 172.21.15.113:0/693658250 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbb7fe700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc06b790 0x7f6bbc0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bbc0cdb70 0x7f6bbc06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 -- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc06b790 msgr2=0x7f6bbc0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc06b790 0x7f6bbc0c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 -- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bbc0cdb70 msgr2=0x7f6bbc06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bbc0cdb70 0x7f6bbc06b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 -- 172.21.15.113:0/693658250 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0c8cf0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bbaffd700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6bbc0c7070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00072c0 tx=0x7f6bb0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00077a0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb000f8d0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000fc80 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc0c8fd0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.826+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc0c7790 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.827+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc0c91c0 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.827+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6bb001d460 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.828+0000 7f6ba37fe700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6ba806b350 0x7f6ba806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.828+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bb0089a70 con 0x7f6ba4003fb0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.828+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6ba806b350 0x7f6ba806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.828+0000 7f6bba7fc700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6ba806b350 0x7f6ba806d810 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bac005040 tx=0x7f6bac009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:13.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:13.832+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6bb00548e0 con 0x7f6ba4003fb0 2024-01-16T13:52:14.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.018+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6bbc0b8870 con 0x7f6ba806b350 2024-01-16T13:52:14.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.025+0000 7f6ba37fe700 1 -- 172.21.15.113:0/693658250 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264606 (secure 0 0 0) 0x7f6bac001290 con 0x7f6ba806b350 2024-01-16T13:52:14.011 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6ba806b350 msgr2=0x7f6ba806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6ba806b350 0x7f6ba806d810 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bac005040 tx=0x7f6bac009030 comp rx=0 tx=0).stop 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 msgr2=0x7f6bbc0c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6bbc0c7070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00072c0 tx=0x7f6bb0007680 comp rx=0 tx=0).stop 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 shutdown_connections 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6ba806b350 0x7f6ba806d810 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bbc06b790 0x7f6bbc0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ba4003fb0 0x7f6bbc0c7070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 --2- 172.21.15.113:0/693658250 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bbc0cdb70 0x7f6bbc06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.028+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 >> 172.21.15.113:0/693658250 conn(0x7f6bbc0d0950 msgr2=0x7f6bbc0b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:14.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.029+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 shutdown_connections 2024-01-16T13:52:14.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.029+0000 7f6bc15d1700 1 -- 172.21.15.113:0/693658250 wait complete. 2024-01-16T13:52:14.014 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:52:14.023 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-01-16T13:52:12.403093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001562"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383364+0000","last_change":"2024-01-16T13:52:04.933545+0000","last_active":"2024-01-16T13:52:06.383364+0000","last_peered":"2024-01-16T13:52:06.383364+0000","last_clean":"2024-01-16T13:52:06.383364+0000","last_became_active":"2024-01-16T13:52:04.932314+0000","last_became_peered":"2024-01-16T13:52:04.932314+0000","last_unstale":"2024-01-16T13:52:06.383364+0000","last_undegraded":"2024-01-16T13:52:06.383364+0000","last_fullsized":"2024-01-16T13:52:06.383364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.917385+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.916958+0000","last_became_peered":"2024-01-16T13:52:04.916958+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382533+0000","last_change":"2024-01-16T13:52:04.917514+0000","last_active":"2024-01-16T13:52:06.382533+0000","last_peered":"2024-01-16T13:52:06.382533+0000","last_clean":"2024-01-16T13:52:06.382533+0000","last_became_active":"2024-01-16T13:52:04.916822+0000","last_became_peered":"2024-01-16T13:52:04.916822+0000","last_unstale":"2024-01-16T13:52:06.382533+0000","last_undegraded":"2024-01-16T13:52:06.382533+0000","last_fullsized":"2024-01-16T13:52:06.382533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:05.397603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904528+0000","last_change":"2024-01-16T13:52:04.926700+0000","last_active":"2024-01-16T13:52:05.904528+0000","last_peered":"2024-01-16T13:52:05.904528+0000","last_clean":"2024-01-16T13:52:05.904528+0000","last_became_active":"2024-01-16T13:52:04.923728+0000","last_became_peered":"2024-01-16T13:52:04.923728+0000","last_unstale":"2024-01-16T13:52:05.904528+0000","last_undegraded":"2024-01-16T13:52:05.904528+0000","last_fullsized":"2024-01-16T13:52:05.904528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382807+0000","last_change":"2024-01-16T13:52:04.931697+0000","last_active":"2024-01-16T13:52:06.382807+0000","last_peered":"2024-01-16T13:52:06.382807+0000","last_clean":"2024-01-16T13:52:06.382807+0000","last_became_active":"2024-01-16T13:52:04.931415+0000","last_became_peered":"2024-01-16T13:52:04.931415+0000","last_unstale":"2024-01-16T13:52:06.382807+0000","last_undegraded":"2024-01-16T13:52:06.382807+0000","last_fullsized":"2024-01-16T13:52:06.382807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904991+0000","last_change":"2024-01-16T13:52:04.925527+0000","last_active":"2024-01-16T13:52:05.904991+0000","last_peered":"2024-01-16T13:52:05.904991+0000","last_clean":"2024-01-16T13:52:05.904991+0000","last_became_active":"2024-01-16T13:52:04.923026+0000","last_became_peered":"2024-01-16T13:52:04.923026+0000","last_unstale":"2024-01-16T13:52:05.904991+0000","last_undegraded":"2024-01-16T13:52:05.904991+0000","last_fullsized":"2024-01-16T13:52:05.904991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382610+0000","last_change":"2024-01-16T13:52:04.930095+0000","last_active":"2024-01-16T13:52:06.382610+0000","last_peered":"2024-01-16T13:52:06.382610+0000","last_clean":"2024-01-16T13:52:06.382610+0000","last_became_active":"2024-01-16T13:52:04.926632+0000","last_became_peered":"2024-01-16T13:52:04.926632+0000","last_unstale":"2024-01-16T13:52:06.382610+0000","last_undegraded":"2024-01-16T13:52:06.382610+0000","last_fullsized":"2024-01-16T13:52:06.382610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382842+0000","last_change":"2024-01-16T13:52:04.927129+0000","last_active":"2024-01-16T13:52:06.382842+0000","last_peered":"2024-01-16T13:52:06.382842+0000","last_clean":"2024-01-16T13:52:06.382842+0000","last_became_active":"2024-01-16T13:52:04.926541+0000","last_became_peered":"2024-01-16T13:52:04.926541+0000","last_unstale":"2024-01-16T13:52:06.382842+0000","last_undegraded":"2024-01-16T13:52:06.382842+0000","last_fullsized":"2024-01-16T13:52:06.382842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383203+0000","last_change":"2024-01-16T13:52:04.935989+0000","last_active":"2024-01-16T13:52:06.383203+0000","last_peered":"2024-01-16T13:52:06.383203+0000","last_clean":"2024-01-16T13:52:06.383203+0000","last_became_active":"2024-01-16T13:52:04.935750+0000","last_became_peered":"2024-01-16T13:52:04.935750+0000","last_unstale":"2024-01-16T13:52:06.383203+0000","last_undegraded":"2024-01-16T13:52:06.383203+0000","last_fullsized":"2024-01-16T13:52:06.383203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904269+0000","last_change":"2024-01-16T13:52:04.941805+0000","last_active":"2024-01-16T13:52:05.904269+0000","last_peered":"2024-01-16T13:52:05.904269+0000","last_clean":"2024-01-16T13:52:05.904269+0000","last_became_active":"2024-01-16T13:52:04.941552+0000","last_became_peered":"2024-01-16T13:52:04.941552+0000","last_unstale":"2024-01-16T13:52:05.904269+0000","last_undegraded":"2024-01-16T13:52:05.904269+0000","last_fullsized":"2024-01-16T13:52:05.904269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904181+0000","last_change":"2024-01-16T13:52:04.941357+0000","last_active":"2024-01-16T13:52:05.904181+0000","last_peered":"2024-01-16T13:52:05.904181+0000","last_clean":"2024-01-16T13:52:05.904181+0000","last_became_active":"2024-01-16T13:52:04.940946+0000","last_became_peered":"2024-01-16T13:52:04.940946+0000","last_unstale":"2024-01-16T13:52:05.904181+0000","last_undegraded":"2024-01-16T13:52:05.904181+0000","last_fullsized":"2024-01-16T13:52:05.904181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383260+0000","last_change":"2024-01-16T13:52:04.923980+0000","last_active":"2024-01-16T13:52:06.383260+0000","last_peered":"2024-01-16T13:52:06.383260+0000","last_clean":"2024-01-16T13:52:06.383260+0000","last_became_active":"2024-01-16T13:52:04.923398+0000","last_became_peered":"2024-01-16T13:52:04.923398+0000","last_unstale":"2024-01-16T13:52:06.383260+0000","last_undegraded":"2024-01-16T13:52:06.383260+0000","last_fullsized":"2024-01-16T13:52:06.383260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383179+0000","last_change":"2024-01-16T13:52:04.922440+0000","last_active":"2024-01-16T13:52:06.383179+0000","last_peered":"2024-01-16T13:52:06.383179+0000","last_clean":"2024-01-16T13:52:06.383179+0000","last_became_active":"2024-01-16T13:52:04.922170+0000","last_became_peered":"2024-01-16T13:52:04.922170+0000","last_unstale":"2024-01-16T13:52:06.383179+0000","last_undegraded":"2024-01-16T13:52:06.383179+0000","last_fullsized":"2024-01-16T13:52:06.383179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382935+0000","last_change":"2024-01-16T13:52:04.918126+0000","last_active":"2024-01-16T13:52:06.382935+0000","last_peered":"2024-01-16T13:52:06.382935+0000","last_clean":"2024-01-16T13:52:06.382935+0000","last_became_active":"2024-01-16T13:52:04.917683+0000","last_became_peered":"2024-01-16T13:52:04.917683+0000","last_unstale":"2024-01-16T13:52:06.382935+0000","last_undegraded":"2024-01-16T13:52:06.382935+0000","last_fullsized":"2024-01-16T13:52:06.382935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904662+0000","last_change":"2024-01-16T13:52:04.919145+0000","last_active":"2024-01-16T13:52:05.904662+0000","last_peered":"2024-01-16T13:52:05.904662+0000","last_clean":"2024-01-16T13:52:05.904662+0000","last_became_active":"2024-01-16T13:52:04.918897+0000","last_became_peered":"2024-01-16T13:52:04.918897+0000","last_unstale":"2024-01-16T13:52:05.904662+0000","last_undegraded":"2024-01-16T13:52:05.904662+0000","last_fullsized":"2024-01-16T13:52:05.904662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904946+0000","last_change":"2024-01-16T13:52:04.923697+0000","last_active":"2024-01-16T13:52:05.904946+0000","last_peered":"2024-01-16T13:52:05.904946+0000","last_clean":"2024-01-16T13:52:05.904946+0000","last_became_active":"2024-01-16T13:52:04.923053+0000","last_became_peered":"2024-01-16T13:52:04.923053+0000","last_unstale":"2024-01-16T13:52:05.904946+0000","last_undegraded":"2024-01-16T13:52:05.904946+0000","last_fullsized":"2024-01-16T13:52:05.904946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904261+0000","last_change":"2024-01-16T13:52:04.927671+0000","last_active":"2024-01-16T13:52:05.904261+0000","last_peered":"2024-01-16T13:52:05.904261+0000","last_clean":"2024-01-16T13:52:05.904261+0000","last_became_active":"2024-01-16T13:52:04.927402+0000","last_became_peered":"2024-01-16T13:52:04.927402+0000","last_unstale":"2024-01-16T13:52:05.904261+0000","last_undegraded":"2024-01-16T13:52:05.904261+0000","last_fullsized":"2024-01-16T13:52:05.904261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382408+0000","last_change":"2024-01-16T13:52:04.924972+0000","last_active":"2024-01-16T13:52:06.382408+0000","last_peered":"2024-01-16T13:52:06.382408+0000","last_clean":"2024-01-16T13:52:06.382408+0000","last_became_active":"2024-01-16T13:52:04.924756+0000","last_became_peered":"2024-01-16T13:52:04.924756+0000","last_unstale":"2024-01-16T13:52:06.382408+0000","last_undegraded":"2024-01-16T13:52:06.382408+0000","last_fullsized":"2024-01-16T13:52:06.382408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:08.127578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383026+0000","last_change":"2024-01-16T13:52:04.932352+0000","last_active":"2024-01-16T13:52:06.383026+0000","last_peered":"2024-01-16T13:52:06.383026+0000","last_clean":"2024-01-16T13:52:06.383026+0000","last_became_active":"2024-01-16T13:52:04.926615+0000","last_became_peered":"2024-01-16T13:52:04.926615+0000","last_unstale":"2024-01-16T13:52:06.383026+0000","last_undegraded":"2024-01-16T13:52:06.383026+0000","last_fullsized":"2024-01-16T13:52:06.383026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382783+0000","last_change":"2024-01-16T13:52:03.906795+0000","last_active":"2024-01-16T13:52:06.382783+0000","last_peered":"2024-01-16T13:52:06.382783+0000","last_clean":"2024-01-16T13:52:06.382783+0000","last_became_active":"2024-01-16T13:52:03.906413+0000","last_became_peered":"2024-01-16T13:52:03.906413+0000","last_unstale":"2024-01-16T13:52:06.382783+0000","last_undegraded":"2024-01-16T13:52:06.382783+0000","last_fullsized":"2024-01-16T13:52:06.382783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383205+0000","last_change":"2024-01-16T13:52:04.922947+0000","last_active":"2024-01-16T13:52:06.383205+0000","last_peered":"2024-01-16T13:52:06.383205+0000","last_clean":"2024-01-16T13:52:06.383205+0000","last_became_active":"2024-01-16T13:52:04.922641+0000","last_became_peered":"2024-01-16T13:52:04.922641+0000","last_unstale":"2024-01-16T13:52:06.383205+0000","last_undegraded":"2024-01-16T13:52:06.383205+0000","last_fullsized":"2024-01-16T13:52:06.383205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383275+0000","last_change":"2024-01-16T13:52:03.911364+0000","last_active":"2024-01-16T13:52:06.383275+0000","last_peered":"2024-01-16T13:52:06.383275+0000","last_clean":"2024-01-16T13:52:06.383275+0000","last_became_active":"2024-01-16T13:52:03.911096+0000","last_became_peered":"2024-01-16T13:52:03.911096+0000","last_unstale":"2024-01-16T13:52:06.383275+0000","last_undegraded":"2024-01-16T13:52:06.383275+0000","last_fullsized":"2024-01-16T13:52:06.383275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382855+0000","last_change":"2024-01-16T13:52:04.941583+0000","last_active":"2024-01-16T13:52:06.382855+0000","last_peered":"2024-01-16T13:52:06.382855+0000","last_clean":"2024-01-16T13:52:06.382855+0000","last_became_active":"2024-01-16T13:52:04.941270+0000","last_became_peered":"2024-01-16T13:52:04.941270+0000","last_unstale":"2024-01-16T13:52:06.382855+0000","last_undegraded":"2024-01-16T13:52:06.382855+0000","last_fullsized":"2024-01-16T13:52:06.382855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383462+0000","last_change":"2024-01-16T13:52:03.908013+0000","last_active":"2024-01-16T13:52:06.383462+0000","last_peered":"2024-01-16T13:52:06.383462+0000","last_clean":"2024-01-16T13:52:06.383462+0000","last_became_active":"2024-01-16T13:52:03.907716+0000","last_became_peered":"2024-01-16T13:52:03.907716+0000","last_unstale":"2024-01-16T13:52:06.383462+0000","last_undegraded":"2024-01-16T13:52:06.383462+0000","last_fullsized":"2024-01-16T13:52:06.383462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904017+0000","last_change":"2024-01-16T13:52:04.920391+0000","last_active":"2024-01-16T13:52:05.904017+0000","last_peered":"2024-01-16T13:52:05.904017+0000","last_clean":"2024-01-16T13:52:05.904017+0000","last_became_active":"2024-01-16T13:52:04.920118+0000","last_became_peered":"2024-01-16T13:52:04.920118+0000","last_unstale":"2024-01-16T13:52:05.904017+0000","last_undegraded":"2024-01-16T13:52:05.904017+0000","last_fullsized":"2024-01-16T13:52:05.904017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382906+0000","last_change":"2024-01-16T13:52:03.905310+0000","last_active":"2024-01-16T13:52:06.382906+0000","last_peered":"2024-01-16T13:52:06.382906+0000","last_clean":"2024-01-16T13:52:06.382906+0000","last_became_active":"2024-01-16T13:52:03.905031+0000","last_became_peered":"2024-01-16T13:52:03.905031+0000","last_unstale":"2024-01-16T13:52:06.382906+0000","last_undegraded":"2024-01-16T13:52:06.382906+0000","last_fullsized":"2024-01-16T13:52:06.382906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904705+0000","last_change":"2024-01-16T13:52:04.912084+0000","last_active":"2024-01-16T13:52:05.904705+0000","last_peered":"2024-01-16T13:52:05.904705+0000","last_clean":"2024-01-16T13:52:05.904705+0000","last_became_active":"2024-01-16T13:52:04.911792+0000","last_became_peered":"2024-01-16T13:52:04.911792+0000","last_unstale":"2024-01-16T13:52:05.904705+0000","last_undegraded":"2024-01-16T13:52:05.904705+0000","last_fullsized":"2024-01-16T13:52:05.904705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:15.505738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383224+0000","last_change":"2024-01-16T13:52:03.912985+0000","last_active":"2024-01-16T13:52:06.383224+0000","last_peered":"2024-01-16T13:52:06.383224+0000","last_clean":"2024-01-16T13:52:06.383224+0000","last_became_active":"2024-01-16T13:52:03.912758+0000","last_became_peered":"2024-01-16T13:52:03.912758+0000","last_unstale":"2024-01-16T13:52:06.383224+0000","last_undegraded":"2024-01-16T13:52:06.383224+0000","last_fullsized":"2024-01-16T13:52:06.383224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382525+0000","last_change":"2024-01-16T13:52:04.917347+0000","last_active":"2024-01-16T13:52:06.382525+0000","last_peered":"2024-01-16T13:52:06.382525+0000","last_clean":"2024-01-16T13:52:06.382525+0000","last_became_active":"2024-01-16T13:52:04.916918+0000","last_became_peered":"2024-01-16T13:52:04.916918+0000","last_unstale":"2024-01-16T13:52:06.382525+0000","last_undegraded":"2024-01-16T13:52:06.382525+0000","last_fullsized":"2024-01-16T13:52:06.382525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:51.832998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903890+0000","last_change":"2024-01-16T13:52:03.903840+0000","last_active":"2024-01-16T13:52:05.903890+0000","last_peered":"2024-01-16T13:52:05.903890+0000","last_clean":"2024-01-16T13:52:05.903890+0000","last_became_active":"2024-01-16T13:52:03.903524+0000","last_became_peered":"2024-01-16T13:52:03.903524+0000","last_unstale":"2024-01-16T13:52:05.903890+0000","last_undegraded":"2024-01-16T13:52:05.903890+0000","last_fullsized":"2024-01-16T13:52:05.903890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:04:07.123155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383068+0000","last_change":"2024-01-16T13:52:04.924992+0000","last_active":"2024-01-16T13:52:06.383068+0000","last_peered":"2024-01-16T13:52:06.383068+0000","last_clean":"2024-01-16T13:52:06.383068+0000","last_became_active":"2024-01-16T13:52:04.924689+0000","last_became_peered":"2024-01-16T13:52:04.924689+0000","last_unstale":"2024-01-16T13:52:06.383068+0000","last_undegraded":"2024-01-16T13:52:06.383068+0000","last_fullsized":"2024-01-16T13:52:06.383068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383042+0000","last_change":"2024-01-16T13:52:03.912658+0000","last_active":"2024-01-16T13:52:06.383042+0000","last_peered":"2024-01-16T13:52:06.383042+0000","last_clean":"2024-01-16T13:52:06.383042+0000","last_became_active":"2024-01-16T13:52:03.912382+0000","last_became_peered":"2024-01-16T13:52:03.912382+0000","last_unstale":"2024-01-16T13:52:06.383042+0000","last_undegraded":"2024-01-16T13:52:06.383042+0000","last_fullsized":"2024-01-16T13:52:06.383042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383509+0000","last_change":"2024-01-16T13:52:04.920332+0000","last_active":"2024-01-16T13:52:06.383509+0000","last_peered":"2024-01-16T13:52:06.383509+0000","last_clean":"2024-01-16T13:52:06.383509+0000","last_became_active":"2024-01-16T13:52:04.920061+0000","last_became_peered":"2024-01-16T13:52:04.920061+0000","last_unstale":"2024-01-16T13:52:06.383509+0000","last_undegraded":"2024-01-16T13:52:06.383509+0000","last_fullsized":"2024-01-16T13:52:06.383509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:53.751325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904985+0000","last_change":"2024-01-16T13:52:03.897543+0000","last_active":"2024-01-16T13:52:05.904985+0000","last_peered":"2024-01-16T13:52:05.904985+0000","last_clean":"2024-01-16T13:52:05.904985+0000","last_became_active":"2024-01-16T13:52:03.897059+0000","last_became_peered":"2024-01-16T13:52:03.897059+0000","last_unstale":"2024-01-16T13:52:05.904985+0000","last_undegraded":"2024-01-16T13:52:05.904985+0000","last_fullsized":"2024-01-16T13:52:05.904985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904732+0000","last_change":"2024-01-16T13:52:04.906828+0000","last_active":"2024-01-16T13:52:05.904732+0000","last_peered":"2024-01-16T13:52:05.904732+0000","last_clean":"2024-01-16T13:52:05.904732+0000","last_became_active":"2024-01-16T13:52:04.906549+0000","last_became_peered":"2024-01-16T13:52:04.906549+0000","last_unstale":"2024-01-16T13:52:05.904732+0000","last_undegraded":"2024-01-16T13:52:05.904732+0000","last_fullsized":"2024-01-16T13:52:05.904732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904051+0000","last_change":"2024-01-16T13:52:03.901711+0000","last_active":"2024-01-16T13:52:05.904051+0000","last_peered":"2024-01-16T13:52:05.904051+0000","last_clean":"2024-01-16T13:52:05.904051+0000","last_became_active":"2024-01-16T13:52:03.901420+0000","last_became_peered":"2024-01-16T13:52:03.901420+0000","last_unstale":"2024-01-16T13:52:05.904051+0000","last_undegraded":"2024-01-16T13:52:05.904051+0000","last_fullsized":"2024-01-16T13:52:05.904051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904698+0000","last_change":"2024-01-16T13:52:04.935326+0000","last_active":"2024-01-16T13:52:05.904698+0000","last_peered":"2024-01-16T13:52:05.904698+0000","last_clean":"2024-01-16T13:52:05.904698+0000","last_became_active":"2024-01-16T13:52:04.934628+0000","last_became_peered":"2024-01-16T13:52:04.934628+0000","last_unstale":"2024-01-16T13:52:05.904698+0000","last_undegraded":"2024-01-16T13:52:05.904698+0000","last_fullsized":"2024-01-16T13:52:05.904698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:30.727607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904576+0000","last_change":"2024-01-16T13:52:03.902616+0000","last_active":"2024-01-16T13:52:05.904576+0000","last_peered":"2024-01-16T13:52:05.904576+0000","last_clean":"2024-01-16T13:52:05.904576+0000","last_became_active":"2024-01-16T13:52:03.902333+0000","last_became_peered":"2024-01-16T13:52:03.902333+0000","last_unstale":"2024-01-16T13:52:05.904576+0000","last_undegraded":"2024-01-16T13:52:05.904576+0000","last_fullsized":"2024-01-16T13:52:05.904576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383250+0000","last_change":"2024-01-16T13:52:04.936172+0000","last_active":"2024-01-16T13:52:06.383250+0000","last_peered":"2024-01-16T13:52:06.383250+0000","last_clean":"2024-01-16T13:52:06.383250+0000","last_became_active":"2024-01-16T13:52:04.935881+0000","last_became_peered":"2024-01-16T13:52:04.935881+0000","last_unstale":"2024-01-16T13:52:06.383250+0000","last_undegraded":"2024-01-16T13:52:06.383250+0000","last_fullsized":"2024-01-16T13:52:06.383250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904808+0000","last_change":"2024-01-16T13:52:03.896280+0000","last_active":"2024-01-16T13:52:05.904808+0000","last_peered":"2024-01-16T13:52:05.904808+0000","last_clean":"2024-01-16T13:52:05.904808+0000","last_became_active":"2024-01-16T13:52:03.896001+0000","last_became_peered":"2024-01-16T13:52:03.896001+0000","last_unstale":"2024-01-16T13:52:05.904808+0000","last_undegraded":"2024-01-16T13:52:05.904808+0000","last_fullsized":"2024-01-16T13:52:05.904808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:51.413262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383116+0000","last_change":"2024-01-16T13:52:04.927368+0000","last_active":"2024-01-16T13:52:06.383116+0000","last_peered":"2024-01-16T13:52:06.383116+0000","last_clean":"2024-01-16T13:52:06.383116+0000","last_became_active":"2024-01-16T13:52:04.927100+0000","last_became_peered":"2024-01-16T13:52:04.927100+0000","last_unstale":"2024-01-16T13:52:06.383116+0000","last_undegraded":"2024-01-16T13:52:06.383116+0000","last_fullsized":"2024-01-16T13:52:06.383116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904369+0000","last_change":"2024-01-16T13:52:03.910360+0000","last_active":"2024-01-16T13:52:05.904369+0000","last_peered":"2024-01-16T13:52:05.904369+0000","last_clean":"2024-01-16T13:52:05.904369+0000","last_became_active":"2024-01-16T13:52:03.910098+0000","last_became_peered":"2024-01-16T13:52:03.910098+0000","last_unstale":"2024-01-16T13:52:05.904369+0000","last_undegraded":"2024-01-16T13:52:05.904369+0000","last_fullsized":"2024-01-16T13:52:05.904369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:41:54.450744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383195+0000","last_change":"2024-01-16T13:52:04.932597+0000","last_active":"2024-01-16T13:52:06.383195+0000","last_peered":"2024-01-16T13:52:06.383195+0000","last_clean":"2024-01-16T13:52:06.383195+0000","last_became_active":"2024-01-16T13:52:04.930511+0000","last_became_peered":"2024-01-16T13:52:04.930511+0000","last_unstale":"2024-01-16T13:52:06.383195+0000","last_undegraded":"2024-01-16T13:52:06.383195+0000","last_fullsized":"2024-01-16T13:52:06.383195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:12:09.520699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382997+0000","last_change":"2024-01-16T13:52:03.906336+0000","last_active":"2024-01-16T13:52:06.382997+0000","last_peered":"2024-01-16T13:52:06.382997+0000","last_clean":"2024-01-16T13:52:06.382997+0000","last_became_active":"2024-01-16T13:52:03.905656+0000","last_became_peered":"2024-01-16T13:52:03.905656+0000","last_unstale":"2024-01-16T13:52:06.382997+0000","last_undegraded":"2024-01-16T13:52:06.382997+0000","last_fullsized":"2024-01-16T13:52:06.382997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382700+0000","last_change":"2024-01-16T13:52:04.927208+0000","last_active":"2024-01-16T13:52:06.382700+0000","last_peered":"2024-01-16T13:52:06.382700+0000","last_clean":"2024-01-16T13:52:06.382700+0000","last_became_active":"2024-01-16T13:52:04.926931+0000","last_became_peered":"2024-01-16T13:52:04.926931+0000","last_unstale":"2024-01-16T13:52:06.382700+0000","last_undegraded":"2024-01-16T13:52:06.382700+0000","last_fullsized":"2024-01-16T13:52:06.382700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905228+0000","last_change":"2024-01-16T13:52:03.911290+0000","last_active":"2024-01-16T13:52:05.905228+0000","last_peered":"2024-01-16T13:52:05.905228+0000","last_clean":"2024-01-16T13:52:05.905228+0000","last_became_active":"2024-01-16T13:52:03.911029+0000","last_became_peered":"2024-01-16T13:52:03.911029+0000","last_unstale":"2024-01-16T13:52:05.905228+0000","last_undegraded":"2024-01-16T13:52:05.905228+0000","last_fullsized":"2024-01-16T13:52:05.905228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:40.789717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383244+0000","last_change":"2024-01-16T13:52:04.932349+0000","last_active":"2024-01-16T13:52:06.383244+0000","last_peered":"2024-01-16T13:52:06.383244+0000","last_clean":"2024-01-16T13:52:06.383244+0000","last_became_active":"2024-01-16T13:52:04.931854+0000","last_became_peered":"2024-01-16T13:52:04.931854+0000","last_unstale":"2024-01-16T13:52:06.383244+0000","last_undegraded":"2024-01-16T13:52:06.383244+0000","last_fullsized":"2024-01-16T13:52:06.383244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:33:51.386065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903653+0000","last_change":"2024-01-16T13:52:03.903815+0000","last_active":"2024-01-16T13:52:05.903653+0000","last_peered":"2024-01-16T13:52:05.903653+0000","last_clean":"2024-01-16T13:52:05.903653+0000","last_became_active":"2024-01-16T13:52:03.903487+0000","last_became_peered":"2024-01-16T13:52:03.903487+0000","last_unstale":"2024-01-16T13:52:05.903653+0000","last_undegraded":"2024-01-16T13:52:05.903653+0000","last_fullsized":"2024-01-16T13:52:05.903653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:49.732984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903921+0000","last_change":"2024-01-16T13:52:04.922443+0000","last_active":"2024-01-16T13:52:05.903921+0000","last_peered":"2024-01-16T13:52:05.903921+0000","last_clean":"2024-01-16T13:52:05.903921+0000","last_became_active":"2024-01-16T13:52:04.922131+0000","last_became_peered":"2024-01-16T13:52:04.922131+0000","last_unstale":"2024-01-16T13:52:05.903921+0000","last_undegraded":"2024-01-16T13:52:05.903921+0000","last_fullsized":"2024-01-16T13:52:05.903921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382223+0000","last_change":"2024-01-16T13:52:03.910304+0000","last_active":"2024-01-16T13:52:06.382223+0000","last_peered":"2024-01-16T13:52:06.382223+0000","last_clean":"2024-01-16T13:52:06.382223+0000","last_became_active":"2024-01-16T13:52:03.910009+0000","last_became_peered":"2024-01-16T13:52:03.910009+0000","last_unstale":"2024-01-16T13:52:06.382223+0000","last_undegraded":"2024-01-16T13:52:06.382223+0000","last_fullsized":"2024-01-16T13:52:06.382223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382979+0000","last_change":"2024-01-16T13:52:04.930880+0000","last_active":"2024-01-16T13:52:06.382979+0000","last_peered":"2024-01-16T13:52:06.382979+0000","last_clean":"2024-01-16T13:52:06.382979+0000","last_became_active":"2024-01-16T13:52:04.930553+0000","last_became_peered":"2024-01-16T13:52:04.930553+0000","last_unstale":"2024-01-16T13:52:06.382979+0000","last_undegraded":"2024-01-16T13:52:06.382979+0000","last_fullsized":"2024-01-16T13:52:06.382979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:40.417449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382946+0000","last_change":"2024-01-16T13:52:03.906573+0000","last_active":"2024-01-16T13:52:06.382946+0000","last_peered":"2024-01-16T13:52:06.382946+0000","last_clean":"2024-01-16T13:52:06.382946+0000","last_became_active":"2024-01-16T13:52:03.906257+0000","last_became_peered":"2024-01-16T13:52:03.906257+0000","last_unstale":"2024-01-16T13:52:06.382946+0000","last_undegraded":"2024-01-16T13:52:06.382946+0000","last_fullsized":"2024-01-16T13:52:06.382946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:14:03.506561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382905+0000","last_change":"2024-01-16T13:52:04.931581+0000","last_active":"2024-01-16T13:52:06.382905+0000","last_peered":"2024-01-16T13:52:06.382905+0000","last_clean":"2024-01-16T13:52:06.382905+0000","last_became_active":"2024-01-16T13:52:04.931306+0000","last_became_peered":"2024-01-16T13:52:04.931306+0000","last_unstale":"2024-01-16T13:52:06.382905+0000","last_undegraded":"2024-01-16T13:52:06.382905+0000","last_fullsized":"2024-01-16T13:52:06.382905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904968+0000","last_change":"2024-01-16T13:52:03.903873+0000","last_active":"2024-01-16T13:52:05.904968+0000","last_peered":"2024-01-16T13:52:05.904968+0000","last_clean":"2024-01-16T13:52:05.904968+0000","last_became_active":"2024-01-16T13:52:03.903582+0000","last_became_peered":"2024-01-16T13:52:03.903582+0000","last_unstale":"2024-01-16T13:52:05.904968+0000","last_undegraded":"2024-01-16T13:52:05.904968+0000","last_fullsized":"2024-01-16T13:52:05.904968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904494+0000","last_change":"2024-01-16T13:52:04.926087+0000","last_active":"2024-01-16T13:52:05.904494+0000","last_peered":"2024-01-16T13:52:05.904494+0000","last_clean":"2024-01-16T13:52:05.904494+0000","last_became_active":"2024-01-16T13:52:04.925736+0000","last_became_peered":"2024-01-16T13:52:04.925736+0000","last_unstale":"2024-01-16T13:52:05.904494+0000","last_undegraded":"2024-01-16T13:52:05.904494+0000","last_fullsized":"2024-01-16T13:52:05.904494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905152+0000","last_change":"2024-01-16T13:52:03.901400+0000","last_active":"2024-01-16T13:52:05.905152+0000","last_peered":"2024-01-16T13:52:05.905152+0000","last_clean":"2024-01-16T13:52:05.905152+0000","last_became_active":"2024-01-16T13:52:03.901087+0000","last_became_peered":"2024-01-16T13:52:03.901087+0000","last_unstale":"2024-01-16T13:52:05.905152+0000","last_undegraded":"2024-01-16T13:52:05.905152+0000","last_fullsized":"2024-01-16T13:52:05.905152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:43.211649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382982+0000","last_change":"2024-01-16T13:52:04.917916+0000","last_active":"2024-01-16T13:52:06.382982+0000","last_peered":"2024-01-16T13:52:06.382982+0000","last_clean":"2024-01-16T13:52:06.382982+0000","last_became_active":"2024-01-16T13:52:04.917649+0000","last_became_peered":"2024-01-16T13:52:04.917649+0000","last_unstale":"2024-01-16T13:52:06.382982+0000","last_undegraded":"2024-01-16T13:52:06.382982+0000","last_fullsized":"2024-01-16T13:52:06.382982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:37.904615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904621+0000","last_change":"2024-01-16T13:52:04.916017+0000","last_active":"2024-01-16T13:52:05.904621+0000","last_peered":"2024-01-16T13:52:05.904621+0000","last_clean":"2024-01-16T13:52:05.904621+0000","last_became_active":"2024-01-16T13:52:04.915765+0000","last_became_peered":"2024-01-16T13:52:04.915765+0000","last_unstale":"2024-01-16T13:52:05.904621+0000","last_undegraded":"2024-01-16T13:52:05.904621+0000","last_fullsized":"2024-01-16T13:52:05.904621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382686+0000","last_change":"2024-01-16T13:52:03.910443+0000","last_active":"2024-01-16T13:52:06.382686+0000","last_peered":"2024-01-16T13:52:06.382686+0000","last_clean":"2024-01-16T13:52:06.382686+0000","last_became_active":"2024-01-16T13:52:03.910025+0000","last_became_peered":"2024-01-16T13:52:03.910025+0000","last_unstale":"2024-01-16T13:52:06.382686+0000","last_undegraded":"2024-01-16T13:52:06.382686+0000","last_fullsized":"2024-01-16T13:52:06.382686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:22.047669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383379+0000","last_change":"2024-01-16T13:52:04.927591+0000","last_active":"2024-01-16T13:52:06.383379+0000","last_peered":"2024-01-16T13:52:06.383379+0000","last_clean":"2024-01-16T13:52:06.383379+0000","last_became_active":"2024-01-16T13:52:04.927020+0000","last_became_peered":"2024-01-16T13:52:04.927020+0000","last_unstale":"2024-01-16T13:52:06.383379+0000","last_undegraded":"2024-01-16T13:52:06.383379+0000","last_fullsized":"2024-01-16T13:52:06.383379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903799+0000","last_change":"2024-01-16T13:52:03.906447+0000","last_active":"2024-01-16T13:52:05.903799+0000","last_peered":"2024-01-16T13:52:05.903799+0000","last_clean":"2024-01-16T13:52:05.903799+0000","last_became_active":"2024-01-16T13:52:03.906082+0000","last_became_peered":"2024-01-16T13:52:03.906082+0000","last_unstale":"2024-01-16T13:52:05.903799+0000","last_undegraded":"2024-01-16T13:52:05.903799+0000","last_fullsized":"2024-01-16T13:52:05.903799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:46:12.841677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-16T13:52:04.913338+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:04.913338+0000","last_peered":"2024-01-16T13:52:04.913338+0000","last_clean":"2024-01-16T13:52:04.913338+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:04.913338+0000","last_undegraded":"2024-01-16T13:52:04.913338+0000","last_fullsized":"2024-01-16T13:52:04.913338+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382590+0000","last_change":"2024-01-16T13:52:04.923544+0000","last_active":"2024-01-16T13:52:06.382590+0000","last_peered":"2024-01-16T13:52:06.382590+0000","last_clean":"2024-01-16T13:52:06.382590+0000","last_became_active":"2024-01-16T13:52:04.923283+0000","last_became_peered":"2024-01-16T13:52:04.923283+0000","last_unstale":"2024-01-16T13:52:06.382590+0000","last_undegraded":"2024-01-16T13:52:06.382590+0000","last_fullsized":"2024-01-16T13:52:06.382590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382524+0000","last_change":"2024-01-16T13:52:03.910495+0000","last_active":"2024-01-16T13:52:06.382524+0000","last_peered":"2024-01-16T13:52:06.382524+0000","last_clean":"2024-01-16T13:52:06.382524+0000","last_became_active":"2024-01-16T13:52:03.910162+0000","last_became_peered":"2024-01-16T13:52:03.910162+0000","last_unstale":"2024-01-16T13:52:06.382524+0000","last_undegraded":"2024-01-16T13:52:06.382524+0000","last_fullsized":"2024-01-16T13:52:06.382524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:44.922338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:03.910098+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:03.909843+0000","last_became_peered":"2024-01-16T13:52:03.909843+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904899+0000","last_change":"2024-01-16T13:52:04.915509+0000","last_active":"2024-01-16T13:52:05.904899+0000","last_peered":"2024-01-16T13:52:05.904899+0000","last_clean":"2024-01-16T13:52:05.904899+0000","last_became_active":"2024-01-16T13:52:04.915240+0000","last_became_peered":"2024-01-16T13:52:04.915240+0000","last_unstale":"2024-01-16T13:52:05.904899+0000","last_undegraded":"2024-01-16T13:52:05.904899+0000","last_fullsized":"2024-01-16T13:52:05.904899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:07.544081+0000","last_change":"2024-01-16T13:52:03.903483+0000","last_active":"2024-01-16T13:52:07.544081+0000","last_peered":"2024-01-16T13:52:07.544081+0000","last_clean":"2024-01-16T13:52:07.544081+0000","last_became_active":"2024-01-16T13:52:03.903147+0000","last_became_peered":"2024-01-16T13:52:03.903147+0000","last_unstale":"2024-01-16T13:52:07.544081+0000","last_undegraded":"2024-01-16T13:52:07.544081+0000","last_fullsized":"2024-01-16T13:52:07.544081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:01.636242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383351+0000","last_change":"2024-01-16T13:52:04.927887+0000","last_active":"2024-01-16T13:52:06.383351+0000","last_peered":"2024-01-16T13:52:06.383351+0000","last_clean":"2024-01-16T13:52:06.383351+0000","last_became_active":"2024-01-16T13:52:04.927626+0000","last_became_peered":"2024-01-16T13:52:04.927626+0000","last_unstale":"2024-01-16T13:52:06.383351+0000","last_undegraded":"2024-01-16T13:52:06.383351+0000","last_fullsized":"2024-01-16T13:52:06.383351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904612+0000","last_change":"2024-01-16T13:52:03.903609+0000","last_active":"2024-01-16T13:52:05.904612+0000","last_peered":"2024-01-16T13:52:05.904612+0000","last_clean":"2024-01-16T13:52:05.904612+0000","last_became_active":"2024-01-16T13:52:03.903258+0000","last_became_peered":"2024-01-16T13:52:03.903258+0000","last_unstale":"2024-01-16T13:52:05.904612+0000","last_undegraded":"2024-01-16T13:52:05.904612+0000","last_fullsized":"2024-01-16T13:52:05.904612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:29.252344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383138+0000","last_change":"2024-01-16T13:52:04.918025+0000","last_active":"2024-01-16T13:52:06.383138+0000","last_peered":"2024-01-16T13:52:06.383138+0000","last_clean":"2024-01-16T13:52:06.383138+0000","last_became_active":"2024-01-16T13:52:04.917716+0000","last_became_peered":"2024-01-16T13:52:04.917716+0000","last_unstale":"2024-01-16T13:52:06.383138+0000","last_undegraded":"2024-01-16T13:52:06.383138+0000","last_fullsized":"2024-01-16T13:52:06.383138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:29.982665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382341+0000","last_change":"2024-01-16T13:52:03.909845+0000","last_active":"2024-01-16T13:52:06.382341+0000","last_peered":"2024-01-16T13:52:06.382341+0000","last_clean":"2024-01-16T13:52:06.382341+0000","last_became_active":"2024-01-16T13:52:03.909573+0000","last_became_peered":"2024-01-16T13:52:03.909573+0000","last_unstale":"2024-01-16T13:52:06.382341+0000","last_undegraded":"2024-01-16T13:52:06.382341+0000","last_fullsized":"2024-01-16T13:52:06.382341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:04.918497+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:04.918198+0000","last_became_peered":"2024-01-16T13:52:04.918198+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:01.775696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904593+0000","last_change":"2024-01-16T13:52:03.904013+0000","last_active":"2024-01-16T13:52:05.904593+0000","last_peered":"2024-01-16T13:52:05.904593+0000","last_clean":"2024-01-16T13:52:05.904593+0000","last_became_active":"2024-01-16T13:52:03.903827+0000","last_became_peered":"2024-01-16T13:52:03.903827+0000","last_unstale":"2024-01-16T13:52:05.904593+0000","last_undegraded":"2024-01-16T13:52:05.904593+0000","last_fullsized":"2024-01-16T13:52:05.904593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:31:50.828576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383603+0000","last_change":"2024-01-16T13:52:04.912859+0000","last_active":"2024-01-16T13:52:06.383603+0000","last_peered":"2024-01-16T13:52:06.383603+0000","last_clean":"2024-01-16T13:52:06.383603+0000","last_became_active":"2024-01-16T13:52:04.912597+0000","last_became_peered":"2024-01-16T13:52:04.912597+0000","last_unstale":"2024-01-16T13:52:06.383603+0000","last_undegraded":"2024-01-16T13:52:06.383603+0000","last_fullsized":"2024-01-16T13:52:06.383603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:13:33.631781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:03.908118+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:03.907800+0000","last_became_peered":"2024-01-16T13:52:03.907800+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.173056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383308+0000","last_change":"2024-01-16T13:52:04.918302+0000","last_active":"2024-01-16T13:52:06.383308+0000","last_peered":"2024-01-16T13:52:06.383308+0000","last_clean":"2024-01-16T13:52:06.383308+0000","last_became_active":"2024-01-16T13:52:04.917994+0000","last_became_peered":"2024-01-16T13:52:04.917994+0000","last_unstale":"2024-01-16T13:52:06.383308+0000","last_undegraded":"2024-01-16T13:52:06.383308+0000","last_fullsized":"2024-01-16T13:52:06.383308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383067+0000","last_change":"2024-01-16T13:52:03.903338+0000","last_active":"2024-01-16T13:52:06.383067+0000","last_peered":"2024-01-16T13:52:06.383067+0000","last_clean":"2024-01-16T13:52:06.383067+0000","last_became_active":"2024-01-16T13:52:03.903053+0000","last_became_peered":"2024-01-16T13:52:03.903053+0000","last_unstale":"2024-01-16T13:52:06.383067+0000","last_undegraded":"2024-01-16T13:52:06.383067+0000","last_fullsized":"2024-01-16T13:52:06.383067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:27.027544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903790+0000","last_change":"2024-01-16T13:52:04.924256+0000","last_active":"2024-01-16T13:52:05.903790+0000","last_peered":"2024-01-16T13:52:05.903790+0000","last_clean":"2024-01-16T13:52:05.903790+0000","last_became_active":"2024-01-16T13:52:04.923945+0000","last_became_peered":"2024-01-16T13:52:04.923945+0000","last_unstale":"2024-01-16T13:52:05.903790+0000","last_undegraded":"2024-01-16T13:52:05.903790+0000","last_fullsized":"2024-01-16T13:52:05.903790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382587+0000","last_change":"2024-01-16T13:52:03.906252+0000","last_active":"2024-01-16T13:52:06.382587+0000","last_peered":"2024-01-16T13:52:06.382587+0000","last_clean":"2024-01-16T13:52:06.382587+0000","last_became_active":"2024-01-16T13:52:03.905949+0000","last_became_peered":"2024-01-16T13:52:03.905949+0000","last_unstale":"2024-01-16T13:52:06.382587+0000","last_undegraded":"2024-01-16T13:52:06.382587+0000","last_fullsized":"2024-01-16T13:52:06.382587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:00.771273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383353+0000","last_change":"2024-01-16T13:52:04.933410+0000","last_active":"2024-01-16T13:52:06.383353+0000","last_peered":"2024-01-16T13:52:06.383353+0000","last_clean":"2024-01-16T13:52:06.383353+0000","last_became_active":"2024-01-16T13:52:04.932914+0000","last_became_peered":"2024-01-16T13:52:04.932914+0000","last_unstale":"2024-01-16T13:52:06.383353+0000","last_undegraded":"2024-01-16T13:52:06.383353+0000","last_fullsized":"2024-01-16T13:52:06.383353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904753+0000","last_change":"2024-01-16T13:52:03.897651+0000","last_active":"2024-01-16T13:52:05.904753+0000","last_peered":"2024-01-16T13:52:05.904753+0000","last_clean":"2024-01-16T13:52:05.904753+0000","last_became_active":"2024-01-16T13:52:03.897380+0000","last_became_peered":"2024-01-16T13:52:03.897380+0000","last_unstale":"2024-01-16T13:52:05.904753+0000","last_undegraded":"2024-01-16T13:52:05.904753+0000","last_fullsized":"2024-01-16T13:52:05.904753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:04.393180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904179+0000","last_change":"2024-01-16T13:52:04.934873+0000","last_active":"2024-01-16T13:52:05.904179+0000","last_peered":"2024-01-16T13:52:05.904179+0000","last_clean":"2024-01-16T13:52:05.904179+0000","last_became_active":"2024-01-16T13:52:04.934573+0000","last_became_peered":"2024-01-16T13:52:04.934573+0000","last_unstale":"2024-01-16T13:52:05.904179+0000","last_undegraded":"2024-01-16T13:52:05.904179+0000","last_fullsized":"2024-01-16T13:52:05.904179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:47:08.128311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382515+0000","last_change":"2024-01-16T13:52:03.911272+0000","last_active":"2024-01-16T13:52:06.382515+0000","last_peered":"2024-01-16T13:52:06.382515+0000","last_clean":"2024-01-16T13:52:06.382515+0000","last_became_active":"2024-01-16T13:52:03.910949+0000","last_became_peered":"2024-01-16T13:52:03.910949+0000","last_unstale":"2024-01-16T13:52:06.382515+0000","last_undegraded":"2024-01-16T13:52:06.382515+0000","last_fullsized":"2024-01-16T13:52:06.382515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:02.878806+0000","last_clean_scrub_stamp":"2024-01-16T13:52:02.878806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:36.900192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383514+0000","last_change":"2024-01-16T13:52:04.933704+0000","last_active":"2024-01-16T13:52:06.383514+0000","last_peered":"2024-01-16T13:52:06.383514+0000","last_clean":"2024-01-16T13:52:06.383514+0000","last_became_active":"2024-01-16T13:52:04.933390+0000","last_became_peered":"2024-01-16T13:52:04.933390+0000","last_unstale":"2024-01-16T13:52:06.383514+0000","last_undegraded":"2024-01-16T13:52:06.383514+0000","last_fullsized":"2024-01-16T13:52:06.383514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.903638+0000","last_change":"2024-01-16T13:52:04.922828+0000","last_active":"2024-01-16T13:52:05.903638+0000","last_peered":"2024-01-16T13:52:05.903638+0000","last_clean":"2024-01-16T13:52:05.903638+0000","last_became_active":"2024-01-16T13:52:04.922428+0000","last_became_peered":"2024-01-16T13:52:04.922428+0000","last_unstale":"2024-01-16T13:52:05.903638+0000","last_undegraded":"2024-01-16T13:52:05.903638+0000","last_fullsized":"2024-01-16T13:52:05.903638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:28.177479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382275+0000","last_change":"2024-01-16T13:52:04.915322+0000","last_active":"2024-01-16T13:52:06.382275+0000","last_peered":"2024-01-16T13:52:06.382275+0000","last_clean":"2024-01-16T13:52:06.382275+0000","last_became_active":"2024-01-16T13:52:04.915031+0000","last_became_peered":"2024-01-16T13:52:04.915031+0000","last_unstale":"2024-01-16T13:52:06.382275+0000","last_undegraded":"2024-01-16T13:52:06.382275+0000","last_fullsized":"2024-01-16T13:52:06.382275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:45.926761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905142+0000","last_change":"2024-01-16T13:52:04.925541+0000","last_active":"2024-01-16T13:52:05.905142+0000","last_peered":"2024-01-16T13:52:05.905142+0000","last_clean":"2024-01-16T13:52:05.905142+0000","last_became_active":"2024-01-16T13:52:04.923411+0000","last_became_peered":"2024-01-16T13:52:04.923411+0000","last_unstale":"2024-01-16T13:52:05.905142+0000","last_undegraded":"2024-01-16T13:52:05.905142+0000","last_fullsized":"2024-01-16T13:52:05.905142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:45:39.789777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382903+0000","last_change":"2024-01-16T13:52:04.919996+0000","last_active":"2024-01-16T13:52:06.382903+0000","last_peered":"2024-01-16T13:52:06.382903+0000","last_clean":"2024-01-16T13:52:06.382903+0000","last_became_active":"2024-01-16T13:52:04.919257+0000","last_became_peered":"2024-01-16T13:52:04.919257+0000","last_unstale":"2024-01-16T13:52:06.382903+0000","last_undegraded":"2024-01-16T13:52:06.382903+0000","last_fullsized":"2024-01-16T13:52:06.382903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:31:44.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382991+0000","last_change":"2024-01-16T13:52:04.921215+0000","last_active":"2024-01-16T13:52:06.382991+0000","last_peered":"2024-01-16T13:52:06.382991+0000","last_clean":"2024-01-16T13:52:06.382991+0000","last_became_active":"2024-01-16T13:52:04.920992+0000","last_became_peered":"2024-01-16T13:52:04.920992+0000","last_unstale":"2024-01-16T13:52:06.382991+0000","last_undegraded":"2024-01-16T13:52:06.382991+0000","last_fullsized":"2024-01-16T13:52:06.382991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:02.640665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904793+0000","last_change":"2024-01-16T13:52:04.911541+0000","last_active":"2024-01-16T13:52:05.904793+0000","last_peered":"2024-01-16T13:52:05.904793+0000","last_clean":"2024-01-16T13:52:05.904793+0000","last_became_active":"2024-01-16T13:52:04.911263+0000","last_became_peered":"2024-01-16T13:52:04.911263+0000","last_unstale":"2024-01-16T13:52:05.904793+0000","last_undegraded":"2024-01-16T13:52:05.904793+0000","last_fullsized":"2024-01-16T13:52:05.904793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904058+0000","last_change":"2024-01-16T13:52:04.932056+0000","last_active":"2024-01-16T13:52:05.904058+0000","last_peered":"2024-01-16T13:52:05.904058+0000","last_clean":"2024-01-16T13:52:05.904058+0000","last_became_active":"2024-01-16T13:52:04.931794+0000","last_became_peered":"2024-01-16T13:52:04.931794+0000","last_unstale":"2024-01-16T13:52:05.904058+0000","last_undegraded":"2024-01-16T13:52:05.904058+0000","last_fullsized":"2024-01-16T13:52:05.904058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:28.613388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383217+0000","last_change":"2024-01-16T13:52:04.918735+0000","last_active":"2024-01-16T13:52:06.383217+0000","last_peered":"2024-01-16T13:52:06.383217+0000","last_clean":"2024-01-16T13:52:06.383217+0000","last_became_active":"2024-01-16T13:52:04.918440+0000","last_became_peered":"2024-01-16T13:52:04.918440+0000","last_unstale":"2024-01-16T13:52:06.383217+0000","last_undegraded":"2024-01-16T13:52:06.383217+0000","last_fullsized":"2024-01-16T13:52:06.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:34:41.794140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905101+0000","last_change":"2024-01-16T13:52:04.923904+0000","last_active":"2024-01-16T13:52:05.905101+0000","last_peered":"2024-01-16T13:52:05.905101+0000","last_clean":"2024-01-16T13:52:05.905101+0000","last_became_active":"2024-01-16T13:52:04.923426+0000","last_became_peered":"2024-01-16T13:52:04.923426+0000","last_unstale":"2024-01-16T13:52:05.905101+0000","last_undegraded":"2024-01-16T13:52:05.905101+0000","last_fullsized":"2024-01-16T13:52:05.905101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:41.030151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.383063+0000","last_change":"2024-01-16T13:52:04.919553+0000","last_active":"2024-01-16T13:52:06.383063+0000","last_peered":"2024-01-16T13:52:06.383063+0000","last_clean":"2024-01-16T13:52:06.383063+0000","last_became_active":"2024-01-16T13:52:04.919053+0000","last_became_peered":"2024-01-16T13:52:04.919053+0000","last_unstale":"2024-01-16T13:52:06.383063+0000","last_undegraded":"2024-01-16T13:52:06.383063+0000","last_fullsized":"2024-01-16T13:52:06.383063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.905086+0000","last_change":"2024-01-16T13:52:04.926131+0000","last_active":"2024-01-16T13:52:05.905086+0000","last_peered":"2024-01-16T13:52:05.905086+0000","last_clean":"2024-01-16T13:52:05.905086+0000","last_became_active":"2024-01-16T13:52:04.925848+0000","last_became_peered":"2024-01-16T13:52:04.925848+0000","last_unstale":"2024-01-16T13:52:05.905086+0000","last_undegraded":"2024-01-16T13:52:05.905086+0000","last_fullsized":"2024-01-16T13:52:05.905086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:37.001707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:06.382745+0000","last_change":"2024-01-16T13:52:04.931968+0000","last_active":"2024-01-16T13:52:06.382745+0000","last_peered":"2024-01-16T13:52:06.382745+0000","last_clean":"2024-01-16T13:52:06.382745+0000","last_became_active":"2024-01-16T13:52:04.931678+0000","last_became_peered":"2024-01-16T13:52:04.931678+0000","last_unstale":"2024-01-16T13:52:06.382745+0000","last_undegraded":"2024-01-16T13:52:06.382745+0000","last_fullsized":"2024-01-16T13:52:06.382745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:20:52.417685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T13:52:05.904558+0000","last_change":"2024-01-16T13:52:04.929411+0000","last_active":"2024-01-16T13:52:05.904558+0000","last_peered":"2024-01-16T13:52:05.904558+0000","last_clean":"2024-01-16T13:52:05.904558+0000","last_became_active":"2024-01-16T13:52:04.929143+0000","last_became_peered":"2024-01-16T13:52:04.929143+0000","last_unstale":"2024-01-16T13:52:05.904558+0000","last_undegraded":"2024-01-16T13:52:05.904558+0000","last_fullsized":"2024-01-16T13:52:05.904558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:03.883229+0000","last_clean_scrub_stamp":"2024-01-16T13:52:03.883229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:28.031967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:52:14.026 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T13:52:14.026 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T13:52:14.026 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T13:52:14.027 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T13:52:14.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.200+0000 7ff9cf6e3700 1 Processor -- start 2024-01-16T13:52:14.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9cf6e3700 1 -- start start 2024-01-16T13:52:14.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9cf6e3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff9c80d2160 0x7ff9c80b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9cf6e3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c81603c0 0x7ff9c80d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9cf6e3700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80d0770 con 0x7ff9c815bed0 2024-01-16T13:52:14.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9cf6e3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c815c2b0 con 0x7ff9c80d2160 2024-01-16T13:52:14.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9cf6e3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80cd100 con 0x7ff9c81603c0 2024-01-16T13:52:14.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.201+0000 7ff9ccc7e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff9c80d2160 0x7ff9c80b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff9c80d2160 0x7ff9c80b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48252/0 (socket says 172.21.15.113:48252) 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 -- 172.21.15.113:0/695787118 learned_addr learned my addr 172.21.15.113:0/695787118 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9cdc80700 1 --2- 172.21.15.113:0/695787118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c81603c0 0x7ff9c80d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 -- 172.21.15.113:0/695787118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c81603c0 msgr2=0x7ff9c80d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 --2- 172.21.15.113:0/695787118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c81603c0 0x7ff9c80d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 -- 172.21.15.113:0/695787118 >> v1:172.21.15.124:6789/0 conn(0x7ff9c815bed0 legacy=0x7ff9c80d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 -- 172.21.15.113:0/695787118 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8160870 con 0x7ff9c80d2160 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9ccc7e700 1 --2- 172.21.15.113:0/695787118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c80d2160 0x7ff9c80b19f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098f0 tx=0x7ff9c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f1883d068aec8b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9bbfff700 1 -- 172.21.15.113:0/695787118 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400b8a0 con 0x7ff9c80d2160 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9bbfff700 1 -- 172.21.15.113:0/695787118 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c400ba00 con 0x7ff9c80d2160 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.202+0000 7ff9bbfff700 1 -- 172.21.15.113:0/695787118 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400bd90 con 0x7ff9c80d2160 2024-01-16T13:52:14.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/695787118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c80d2160 msgr2=0x7ff9c80b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/695787118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c80d2160 0x7ff9c80b19f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098f0 tx=0x7ff9c4009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/695787118 shutdown_connections 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/695787118 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c81603c0 0x7ff9c80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/695787118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c80d2160 0x7ff9c80b19f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/695787118 >> 172.21.15.113:0/695787118 conn(0x7ff9c8073130 msgr2=0x7ff9c809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/695787118 shutdown_connections 2024-01-16T13:52:14.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.203+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/695787118 wait complete. 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.204+0000 7ff9cf6e3700 1 Processor -- start 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.204+0000 7ff9cf6e3700 1 -- start start 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c80d2160 0x7ff9c80c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff9c815bed0 0x7ff9c8159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 0x7ff9c815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80c4f90 con 0x7ff9c81603c0 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8153a40 con 0x7ff9c815bed0 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c806c090 con 0x7ff9c80d2160 2024-01-16T13:52:14.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 0x7ff9c815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 0x7ff9c815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48268/0 (socket says 172.21.15.113:48268) 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 -- 172.21.15.113:0/351940493 learned_addr learned my addr 172.21.15.113:0/351940493 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9ccc7e700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff9c815bed0 0x7ff9c8159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cd47f700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c80d2160 0x7ff9c80c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 -- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c80d2160 msgr2=0x7ff9c80c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c80d2160 0x7ff9c80c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 -- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff9c815bed0 msgr2=0x7ff9c8159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff9c815bed0 0x7ff9c8159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 -- 172.21.15.113:0/351940493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c4009550 con 0x7ff9c81603c0 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cdc80700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 0x7ff9c815a1f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0007e90 tx=0x7ff9c000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c000ed30 con 0x7ff9c81603c0 2024-01-16T13:52:14.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c000ee90 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0014660 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.205+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c806c370 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.206+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c806c930 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.206+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c8067f60 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.207+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff9c00147c0 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.207+0000 7ff9b9ffb700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ac06b350 0x7ff9ac06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.207+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9c008a0f0 con 0x7ff9c81603c0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.207+0000 7ff9cd47f700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ac06b350 0x7ff9ac06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.208+0000 7ff9cd47f700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ac06b350 0x7ff9ac06d810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc007fd0 tx=0x7ff9bc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.212+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9c00550a0 con 0x7ff9c81603c0 2024-01-16T13:52:14.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.447+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff9c80b7bd0 con 0x7ff9c81603c0 2024-01-16T13:52:14.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.447+0000 7ff9b9ffb700 1 -- 172.21.15.113:0/351940493 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff9c00586c0 con 0x7ff9c81603c0 2024-01-16T13:52:14.429 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:14.429 INFO:teuthology.orchestra.run.smithi113.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T13:52:14.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ac06b350 msgr2=0x7ff9ac06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ac06b350 0x7ff9ac06d810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc007fd0 tx=0x7ff9bc009040 comp rx=0 tx=0).stop 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 msgr2=0x7ff9c815a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 0x7ff9c815a1f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0007e90 tx=0x7ff9c000e3c0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 shutdown_connections 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ac06b350 0x7ff9ac06d810 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9c81603c0 0x7ff9c815a1f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff9c80d2160 0x7ff9c80c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 --2- 172.21.15.113:0/351940493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff9c815bed0 0x7ff9c8159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 >> 172.21.15.113:0/351940493 conn(0x7ff9c8073130 msgr2=0x7ff9c80b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 shutdown_connections 2024-01-16T13:52:14.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.449+0000 7ff9cf6e3700 1 -- 172.21.15.113:0/351940493 wait complete. 2024-01-16T13:52:14.443 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T13:52:14.443 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:14.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 Processor -- start 2024-01-16T13:52:14.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 -- start start 2024-01-16T13:52:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f778c0b3370 0x7f778c1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f778c0d1c70 0x7f778c0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0b22d0 con 0x7f778c15bdd0 2024-01-16T13:52:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c0b2f50 con 0x7f778c0b3370 2024-01-16T13:52:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.583+0000 7f779403e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c0d2640 con 0x7f778c0d1c70 2024-01-16T13:52:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f778c0b3370 0x7f778c1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f778c0b3370 0x7f778c1604e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48280/0 (socket says 172.21.15.113:48280) 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 -- 172.21.15.113:0/2632016269 learned_addr learned my addr 172.21.15.113:0/2632016269 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77925db700 1 --2- 172.21.15.113:0/2632016269 >> v2:172.21.15.124:3300/0 conn(0x7f778c0d1c70 0x7f778c0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 -- 172.21.15.113:0/2632016269 >> v2:172.21.15.124:3300/0 conn(0x7f778c0d1c70 msgr2=0x7f778c0d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 --2- 172.21.15.113:0/2632016269 >> v2:172.21.15.124:3300/0 conn(0x7f778c0d1c70 0x7f778c0d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 -- 172.21.15.113:0/2632016269 >> v1:172.21.15.124:6789/0 conn(0x7f778c15bdd0 legacy=0x7f778c0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 -- 172.21.15.113:0/2632016269 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0b25b0 con 0x7f778c0b3370 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f77915d9700 1 --2- 172.21.15.113:0/2632016269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0b3370 0x7f778c1604e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f778c160a50 tx=0x7f7788009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae3fb402f7c75a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f7790dd8700 1 -- 172.21.15.113:0/2632016269 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788011830 con 0x7f778c0b3370 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f7790dd8700 1 -- 172.21.15.113:0/2632016269 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7788011990 con 0x7f778c0b3370 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.584+0000 7f7790dd8700 1 -- 172.21.15.113:0/2632016269 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788011d20 con 0x7f778c0b3370 2024-01-16T13:52:14.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 -- 172.21.15.113:0/2632016269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0b3370 msgr2=0x7f778c1604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 --2- 172.21.15.113:0/2632016269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0b3370 0x7f778c1604e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f778c160a50 tx=0x7f7788009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 -- 172.21.15.113:0/2632016269 shutdown_connections 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 --2- 172.21.15.113:0/2632016269 >> v2:172.21.15.124:3300/0 conn(0x7f778c0d1c70 0x7f778c0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 --2- 172.21.15.113:0/2632016269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0b3370 0x7f778c1604e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 -- 172.21.15.113:0/2632016269 >> 172.21.15.113:0/2632016269 conn(0x7f778c0731a0 msgr2=0x7f778c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 -- 172.21.15.113:0/2632016269 shutdown_connections 2024-01-16T13:52:14.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.585+0000 7f779403e700 1 -- 172.21.15.113:0/2632016269 wait complete. 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.586+0000 7f779403e700 1 Processor -- start 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.586+0000 7f779403e700 1 -- start start 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f778c0b3370 0x7f778c157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 0x7f778c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f778c15bdd0 0x7f778c0cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0cb6b0 con 0x7f778c0d1c70 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0cbad0 con 0x7f778c15bdd0 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0b76d0 con 0x7f778c0b3370 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 0x7f778c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 0x7f778c0cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48282/0 (socket says 172.21.15.113:48282) 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 -- 172.21.15.113:0/3636759874 learned_addr learned my addr 172.21.15.113:0/3636759874 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f7791dda700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f778c0b3370 0x7f778c157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77925db700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f778c15bdd0 0x7f778c0cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 -- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f778c0b3370 msgr2=0x7f778c157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f778c0b3370 0x7f778c157460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 -- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f778c15bdd0 msgr2=0x7f778c0cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f778c15bdd0 0x7f778c0cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 -- 172.21.15.113:0/3636759874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788009550 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77925db700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f778c15bdd0 0x7f778c0cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77915d9700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 0x7f778c0cd6d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7788000c00 tx=0x7f7788004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788004570 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778801f070 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c0caa30 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.587+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880038e0 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.588+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c0caff0 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.588+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c0cb7f0 con 0x7f778c0d1c70 2024-01-16T13:52:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.589+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7788021030 con 0x7f778c0d1c70 2024-01-16T13:52:14.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.589+0000 7f77827fc700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f777006b350 0x7f777006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.589+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f778808b000 con 0x7f778c0d1c70 2024-01-16T13:52:14.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.589+0000 7f7791dda700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f777006b350 0x7f777006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.590+0000 7f7791dda700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f777006b350 0x7f777006d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f777c000f60 tx=0x7f777c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.594+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f778c0cb7f0 con 0x7f778c0d1c70 2024-01-16T13:52:14.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.785+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f778c0cb7f0 con 0x7f778c0d1c70 2024-01-16T13:52:14.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.786+0000 7f77827fc700 1 -- 172.21.15.113:0/3636759874 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f778c0cb7f0 con 0x7f778c0d1c70 2024-01-16T13:52:14.768 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:14.769 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f777006b350 msgr2=0x7f777006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f777006b350 0x7f777006d810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f777c000f60 tx=0x7f777c009040 comp rx=0 tx=0).stop 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 msgr2=0x7f778c0cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 0x7f778c0cd6d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7788000c00 tx=0x7f7788004750 comp rx=0 tx=0).stop 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 shutdown_connections 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f777006b350 0x7f777006d810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f778c15bdd0 0x7f778c0cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f778c0b3370 0x7f778c157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 --2- 172.21.15.113:0/3636759874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f778c0d1c70 0x7f778c0cd6d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 >> 172.21.15.113:0/3636759874 conn(0x7f778c0731a0 msgr2=0x7f778c06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 shutdown_connections 2024-01-16T13:52:14.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.788+0000 7f779403e700 1 -- 172.21.15.113:0/3636759874 wait complete. 2024-01-16T13:52:14.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:14.783 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:14.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.924+0000 7f01179aa700 1 Processor -- start 2024-01-16T13:52:14.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.924+0000 7f01179aa700 1 -- start start 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f01179aa700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f01100afb50 0x7f011015f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f01179aa700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0110161cc0 0x7f01100b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f01179aa700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01100b5c10 con 0x7f0110073f90 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f01179aa700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011006ee90 con 0x7f01100afb50 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f01179aa700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011015f690 con 0x7f0110161cc0 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f0115746700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f01100afb50 0x7f011015f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f0115746700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f01100afb50 0x7f011015f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48284/0 (socket says 172.21.15.113:48284) 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f0115746700 1 -- 172.21.15.113:0/2254682222 learned_addr learned my addr 172.21.15.113:0/2254682222 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.925+0000 7f0114f45700 1 --2- 172.21.15.113:0/2254682222 >> v2:172.21.15.124:3300/0 conn(0x7f0110161cc0 0x7f01100b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0115746700 1 -- 172.21.15.113:0/2254682222 >> v2:172.21.15.124:3300/0 conn(0x7f0110161cc0 msgr2=0x7f01100b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0115746700 1 --2- 172.21.15.113:0/2254682222 >> v2:172.21.15.124:3300/0 conn(0x7f0110161cc0 0x7f01100b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0115746700 1 -- 172.21.15.113:0/2254682222 >> v1:172.21.15.124:6789/0 conn(0x7f0110073f90 legacy=0x7f0110074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0115746700 1 -- 172.21.15.113:0/2254682222 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01101621f0 con 0x7f01100afb50 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0115746700 1 --2- 172.21.15.113:0/2254682222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01100afb50 0x7f011015f150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01100adf90 tx=0x7f0100009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d55cea923b4292 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0107fff700 1 -- 172.21.15.113:0/2254682222 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100011830 con 0x7f01100afb50 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0107fff700 1 -- 172.21.15.113:0/2254682222 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100011990 con 0x7f01100afb50 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f0107fff700 1 -- 172.21.15.113:0/2254682222 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100011d20 con 0x7f01100afb50 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f01179aa700 1 -- 172.21.15.113:0/2254682222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01100afb50 msgr2=0x7f011015f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.926+0000 7f01179aa700 1 --2- 172.21.15.113:0/2254682222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01100afb50 0x7f011015f150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01100adf90 tx=0x7f0100009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.927+0000 7f01179aa700 1 -- 172.21.15.113:0/2254682222 shutdown_connections 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.927+0000 7f01179aa700 1 --2- 172.21.15.113:0/2254682222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01100afb50 0x7f011015f150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.927+0000 7f01179aa700 1 --2- 172.21.15.113:0/2254682222 >> v2:172.21.15.124:3300/0 conn(0x7f0110161cc0 0x7f01100b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.927+0000 7f01179aa700 1 -- 172.21.15.113:0/2254682222 >> 172.21.15.113:0/2254682222 conn(0x7f0110072a20 msgr2=0x7f011006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.927+0000 7f01179aa700 1 -- 172.21.15.113:0/2254682222 shutdown_connections 2024-01-16T13:52:14.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.927+0000 7f01179aa700 1 -- 172.21.15.113:0/2254682222 wait complete. 2024-01-16T13:52:14.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.928+0000 7f01179aa700 1 Processor -- start 2024-01-16T13:52:14.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.928+0000 7f01179aa700 1 -- start start 2024-01-16T13:52:14.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.928+0000 7f01179aa700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0110073f90 0x7f0110156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.928+0000 7f01179aa700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01100afb50 0x7f01101f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f01179aa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 0x7f01101536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f01179aa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110158f30 con 0x7f0110161cc0 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f01179aa700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01101f7e10 con 0x7f01100afb50 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f01179aa700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0110157480 con 0x7f0110073f90 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 0x7f01101536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 0x7f01101536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48300/0 (socket says 172.21.15.113:48300) 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 -- 172.21.15.113:0/3296188696 learned_addr learned my addr 172.21.15.113:0/3296188696 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0114f45700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01100afb50 0x7f01101f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115746700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0110073f90 0x7f0110156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 -- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0110073f90 msgr2=0x7f0110156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0110073f90 0x7f0110156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 -- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01100afb50 msgr2=0x7f01101f78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01100afb50 0x7f01101f78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:14.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 -- 172.21.15.113:0/3296188696 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100009550 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0115f47700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 0x7f01101536a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f010800cc80 tx=0x7f010800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108003c00 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108003d60 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0110157760 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800aaa0 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.929+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0110159420 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.930+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0110068f10 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.931+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f010800ac00 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.931+0000 7f0105ffb700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f00f406b350 0x7f00f406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.931+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0108089c10 con 0x7f0110161cc0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.931+0000 7f0115746700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f00f406b350 0x7f00f406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:14.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.932+0000 7f0115746700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f00f406b350 0x7f00f406d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0100009c10 tx=0x7f0100008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:14.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:14.936+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01080549e0 con 0x7f0110161cc0 2024-01-16T13:52:15.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.126+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f011015df00 con 0x7f0110161cc0 2024-01-16T13:52:15.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.126+0000 7f0105ffb700 1 -- 172.21.15.113:0/3296188696 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0108058000 con 0x7f0110161cc0 2024-01-16T13:52:15.109 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.128+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f00f406b350 msgr2=0x7f00f406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.128+0000 7f01179aa700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f00f406b350 0x7f00f406d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0100009c10 tx=0x7f0100008030 comp rx=0 tx=0).stop 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 msgr2=0x7f01101536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 0x7f01101536a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f010800cc80 tx=0x7f010800cf90 comp rx=0 tx=0).stop 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 shutdown_connections 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f00f406b350 0x7f00f406d810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0110161cc0 0x7f01101536a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0110073f90 0x7f0110156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 --2- 172.21.15.113:0/3296188696 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01100afb50 0x7f01101f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 >> 172.21.15.113:0/3296188696 conn(0x7f0110072a20 msgr2=0x7f01100ae4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:15.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 shutdown_connections 2024-01-16T13:52:15.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.129+0000 7f01179aa700 1 -- 172.21.15.113:0/3296188696 wait complete. 2024-01-16T13:52:15.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:15.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.264+0000 7ffbcc141700 1 Processor -- start 2024-01-16T13:52:15.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.264+0000 7ffbcc141700 1 -- start start 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbcc141700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ffbc415e110 0x7ffbc406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbcc141700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 0x7ffbc406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbcc141700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40affe0 con 0x7ffbc40afb30 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbcc141700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc406d980 con 0x7ffbc415e110 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbcc141700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40b0260 con 0x7ffbc406e420 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbc96dc700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ffbc40afb30 0x7ffbc406aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39318/0 (socket says 172.21.15.113:39318) 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbc96dc700 1 -- 172.21.15.113:0/2569396197 learned_addr learned my addr 172.21.15.113:0/2569396197 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbc9edd700 1 --2- 172.21.15.113:0/2569396197 >> v2:172.21.15.124:3300/0 conn(0x7ffbc415e110 0x7ffbc406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbca6de700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 0x7ffbc406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817844792 0 0) 0x7ffbc40affe0 con 0x7ffbc40afb30 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.265+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb00035b0 con 0x7ffbc40afb30 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799829515 0 0) 0x7ffbb00035b0 con 0x7ffbc40afb30 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 msgr2=0x7ffbc406e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 0x7ffbc406e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 >> v2:172.21.15.124:3300/0 conn(0x7ffbc415e110 msgr2=0x7ffbc406f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 --2- 172.21.15.113:0/2569396197 >> v2:172.21.15.124:3300/0 conn(0x7ffbc415e110 0x7ffbc406f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40b0540 con 0x7ffbc40afb30 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7ffbc0001670 con 0x7ffbc40afb30 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc9edd700 1 --2- 172.21.15.113:0/2569396197 >> v2:172.21.15.124:3300/0 conn(0x7ffbc415e110 0x7ffbc406f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 >> v1:172.21.15.113:6789/0 conn(0x7ffbc40afb30 legacy=0x7ffbc406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 0x7ffbb0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbb0003f80 0x7ffbb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb0004d90 0x7ffbb0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc406d980 con 0x7ffbb0004d90 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40b0260 con 0x7ffbb0003f80 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb000ba10 con 0x7ffbc406e420 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb0004d90 0x7ffbb0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc9edd700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 0x7ffbb0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc96dc700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbb0003f80 0x7ffbb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 -- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 msgr2=0x7ffbb0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc406e420 0x7ffbb0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 -- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbb0003f80 msgr2=0x7ffbb0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbb0003f80 0x7ffbb0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 -- 172.21.15.113:0/2569396197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb00035b0 con 0x7ffbb0004d90 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbca6de700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb0004d90 0x7ffbb0005210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009fc0 tx=0x7ffbb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e8e29edf4c4df2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00ebd0 con 0x7ffbb0004d90 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc00f040 con 0x7ffbb0004d90 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.266+0000 7ffbc8edb700 1 -- 172.21.15.113:0/2569396197 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc014480 con 0x7ffbb0004d90 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbcc141700 1 -- 172.21.15.113:0/2569396197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb0004d90 msgr2=0x7ffbb0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbcc141700 1 --2- 172.21.15.113:0/2569396197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbb0004d90 0x7ffbb0005210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009fc0 tx=0x7ffbb0005880 comp rx=0 tx=0).stop 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbca6de700 1 -- 172.21.15.113:0/2569396197 reap_dead start 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbcc141700 1 -- 172.21.15.113:0/2569396197 shutdown_connections 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbcc141700 1 -- 172.21.15.113:0/2569396197 >> 172.21.15.113:0/2569396197 conn(0x7ffbc40b3250 msgr2=0x7ffbc40d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbcc141700 1 -- 172.21.15.113:0/2569396197 shutdown_connections 2024-01-16T13:52:15.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.267+0000 7ffbcc141700 1 -- 172.21.15.113:0/2569396197 wait complete. 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.269+0000 7ffbcc141700 1 Processor -- start 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.269+0000 7ffbcc141700 1 -- start start 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.269+0000 7ffbcc141700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc414e440 0x7ffbc40c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.269+0000 7ffbcc141700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbc414e820 0x7ffbc40b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.269+0000 7ffbcc141700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 0x7ffbc40b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.269+0000 7ffbcc141700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40c65a0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbcc141700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40c5c00 con 0x7ffbc414e820 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbcc141700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40c5ee0 con 0x7ffbc414e440 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 0x7ffbc40b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 0x7ffbc40b10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48318/0 (socket says 172.21.15.113:48318) 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 -- 172.21.15.113:0/30198138 learned_addr learned my addr 172.21.15.113:0/30198138 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbc9edd700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc414e440 0x7ffbc40c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbc96dc700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbc414e820 0x7ffbc40b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 -- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc414e440 msgr2=0x7ffbc40c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc414e440 0x7ffbc40c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 -- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbc414e820 msgr2=0x7ffbc40b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbc414e820 0x7ffbc40b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 -- 172.21.15.113:0/30198138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc406b880 con 0x7ffbc40c6f00 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbca6de700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 0x7ffbc40b10f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009f60 tx=0x7ffbbc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc007000 con 0x7ffbc40c6f00 2024-01-16T13:52:15.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc003e70 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc40c61c0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc0159e0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.270+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc41f3bb0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.271+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc40c66e0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.271+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffbbc024070 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.272+0000 7ffbb67fc700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffba806b350 0x7ffba806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.272+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbbc08b2a0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.272+0000 7ffbc9edd700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffba806b350 0x7ffba806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.272+0000 7ffbc9edd700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffba806b350 0x7ffba806d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009e60 tx=0x7ffbb8009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.276+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffbc40c66e0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.467+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbc40c66e0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.467+0000 7ffbb67fc700 1 -- 172.21.15.113:0/30198138 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffbc40c66e0 con 0x7ffbc40c6f00 2024-01-16T13:52:15.449 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffba806b350 msgr2=0x7ffba806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffba806b350 0x7ffba806d810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009e60 tx=0x7ffbb8009210 comp rx=0 tx=0).stop 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 msgr2=0x7ffbc40b10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 0x7ffbc40b10f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009f60 tx=0x7ffbbc009750 comp rx=0 tx=0).stop 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 shutdown_connections 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffba806b350 0x7ffba806d810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbc40c6f00 0x7ffbc40b10f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffbc414e440 0x7ffbc40c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 --2- 172.21.15.113:0/30198138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbc414e820 0x7ffbc40b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 >> 172.21.15.113:0/30198138 conn(0x7ffbc40b3250 msgr2=0x7ffbc406e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 shutdown_connections 2024-01-16T13:52:15.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.469+0000 7ffbcc141700 1 -- 172.21.15.113:0/30198138 wait complete. 2024-01-16T13:52:15.463 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:15.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.605+0000 7fcab2513700 1 Processor -- start 2024-01-16T13:52:15.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.605+0000 7fcab2513700 1 -- start start 2024-01-16T13:52:15.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.605+0000 7fcab2513700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcaac161690 0x7fcaac0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.605+0000 7fcab2513700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.605+0000 7fcab2513700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fcaac0d1730 0x7fcaac09aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcab2513700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaac0d0a70 con 0x7fcaac161690 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcab2513700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaac0d1c10 con 0x7fcaac0d1730 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcab2513700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac0b2ab0 con 0x7fcaac161a70 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcaac161690 0x7fcaac0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcaac161690 0x7fcaac0d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48324/0 (socket says 172.21.15.113:48324) 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 -- 172.21.15.113:0/1382418111 learned_addr learned my addr 172.21.15.113:0/1382418111 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaab7fe700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcab0ab0700 1 --2- 172.21.15.113:0/1382418111 >> v2:172.21.15.124:3300/0 conn(0x7fcaac0d1730 0x7fcaac09aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 -- 172.21.15.113:0/1382418111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 msgr2=0x7fcaac0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 -- 172.21.15.113:0/1382418111 >> v2:172.21.15.124:3300/0 conn(0x7fcaac0d1730 msgr2=0x7fcaac09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 --2- 172.21.15.113:0/1382418111 >> v2:172.21.15.124:3300/0 conn(0x7fcaac0d1730 0x7fcaac09aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 -- 172.21.15.113:0/1382418111 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac0b2d90 con 0x7fcaac161690 2024-01-16T13:52:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcab0ab0700 1 --2- 172.21.15.113:0/1382418111 >> v2:172.21.15.124:3300/0 conn(0x7fcaac0d1730 0x7fcaac09aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaab7fe700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaabfff700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac161690 0x7fcaac0d0f20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcaac072be0 tx=0x7fca9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df141138c0a09aca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaaaffd700 1 -- 172.21.15.113:0/1382418111 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00b870 con 0x7fcaac161690 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaaaffd700 1 -- 172.21.15.113:0/1382418111 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c00b9d0 con 0x7fcaac161690 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.606+0000 7fcaaaffd700 1 -- 172.21.15.113:0/1382418111 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00bd60 con 0x7fcaac161690 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 -- 172.21.15.113:0/1382418111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac161690 msgr2=0x7fcaac0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac161690 0x7fcaac0d0f20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcaac072be0 tx=0x7fca9c009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 -- 172.21.15.113:0/1382418111 shutdown_connections 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac161690 0x7fcaac0d0f20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 --2- 172.21.15.113:0/1382418111 >> v2:172.21.15.124:3300/0 conn(0x7fcaac0d1730 0x7fcaac09aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 --2- 172.21.15.113:0/1382418111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 -- 172.21.15.113:0/1382418111 >> 172.21.15.113:0/1382418111 conn(0x7fcaac0d1e50 msgr2=0x7fcaac0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:15.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 -- 172.21.15.113:0/1382418111 shutdown_connections 2024-01-16T13:52:15.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.607+0000 7fcab2513700 1 -- 172.21.15.113:0/1382418111 wait complete. 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.608+0000 7fcab2513700 1 Processor -- start 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 -- start start 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 0x7fcaac15af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaac161690 0x7fcaac0b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac0b6de0 con 0x7fcaac0d1730 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac0cbcc0 con 0x7fcaac161690 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.609+0000 7fcab2513700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac0b61d0 con 0x7fcaac161a70 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 0x7fcaac15af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 0x7fcaac15af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48334/0 (socket says 172.21.15.113:48334) 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 -- 172.21.15.113:0/1940440973 learned_addr learned my addr 172.21.15.113:0/1940440973 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:15.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcab0ab0700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaab7fe700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaac161690 0x7fcaac0b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 -- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 msgr2=0x7fcaac15b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac15b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 -- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaac161690 msgr2=0x7fcaac0b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaac161690 0x7fcaac0b58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 -- 172.21.15.113:0/1940440973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c009550 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaabfff700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 0x7fcaac15af30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcaac072870 tx=0x7fca9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c004310 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c01f070 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c0036a0 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaac0b64b0 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.610+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac0b6a70 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.611+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac069020 con 0x7fcaac0d1730 2024-01-16T13:52:15.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.612+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fca9c021030 con 0x7fcaac0d1730 2024-01-16T13:52:15.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.612+0000 7fcaa8ff9700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca9806b350 0x7fca9806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.612+0000 7fcaab7fe700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca9806b350 0x7fca9806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.612+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca9c08ae50 con 0x7fcaac0d1730 2024-01-16T13:52:15.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.613+0000 7fcaab7fe700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca9806b350 0x7fca9806d810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0cbfa0 tx=0x7fcaa0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.617+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca9c055d50 con 0x7fcaac0d1730 2024-01-16T13:52:15.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.808+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcaac06baa0 con 0x7fcaac0d1730 2024-01-16T13:52:15.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.808+0000 7fcaa8ff9700 1 -- 172.21.15.113:0/1940440973 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca9c0150a0 con 0x7fcaac0d1730 2024-01-16T13:52:15.790 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:15.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca9806b350 msgr2=0x7fca9806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca9806b350 0x7fca9806d810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0cbfa0 tx=0x7fcaa0008040 comp rx=0 tx=0).stop 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 msgr2=0x7fcaac15af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 0x7fcaac15af30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcaac072870 tx=0x7fca9c004750 comp rx=0 tx=0).stop 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 shutdown_connections 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca9806b350 0x7fca9806d810 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaac161a70 0x7fcaac15b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcaac0d1730 0x7fcaac15af30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 --2- 172.21.15.113:0/1940440973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaac161690 0x7fcaac0b58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 >> 172.21.15.113:0/1940440973 conn(0x7fcaac0d1e50 msgr2=0x7fcaac0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 shutdown_connections 2024-01-16T13:52:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.810+0000 7fcab2513700 1 -- 172.21.15.113:0/1940440973 wait complete. 2024-01-16T13:52:15.804 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:15.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.942+0000 7fe2e173e700 1 Processor -- start 2024-01-16T13:52:15.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.942+0000 7fe2e173e700 1 -- start start 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2e173e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe2dc15bed0 0x7fe2dc0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2e173e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe2dc1603c0 0x7fe2dc0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2e173e700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2dc0d0770 con 0x7fe2dc0d1a50 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2e173e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc15c2b0 con 0x7fe2dc15bed0 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2e173e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2dc0cd100 con 0x7fe2dc1603c0 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2daffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe2dc15bed0 0x7fe2dc0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2daffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe2dc15bed0 0x7fe2dc0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48336/0 (socket says 172.21.15.113:48336) 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2db7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe2dc0d1a50 0x7fe2dc0d1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39334/0 (socket says 172.21.15.113:39334) 2024-01-16T13:52:15.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2daffd700 1 -- 172.21.15.113:0/1264338213 learned_addr learned my addr 172.21.15.113:0/1264338213 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.943+0000 7fe2da7fc700 1 --2- 172.21.15.113:0/1264338213 >> v2:172.21.15.124:3300/0 conn(0x7fe2dc1603c0 0x7fe2dc0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2daffd700 1 -- 172.21.15.113:0/1264338213 >> v2:172.21.15.124:3300/0 conn(0x7fe2dc1603c0 msgr2=0x7fe2dc0d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2daffd700 1 --2- 172.21.15.113:0/1264338213 >> v2:172.21.15.124:3300/0 conn(0x7fe2dc1603c0 0x7fe2dc0d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2daffd700 1 -- 172.21.15.113:0/1264338213 >> v1:172.21.15.113:6789/0 conn(0x7fe2dc0d1a50 legacy=0x7fe2dc0d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2daffd700 1 -- 172.21.15.113:0/1264338213 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0d09a0 con 0x7fe2dc15bed0 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2da7fc700 1 --2- 172.21.15.113:0/1264338213 >> v2:172.21.15.124:3300/0 conn(0x7fe2dc1603c0 0x7fe2dc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2daffd700 1 --2- 172.21.15.113:0/1264338213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0b19f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0d2a30 tx=0x7fe2d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe7af1e47959cb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2d9ffb700 1 -- 172.21.15.113:0/1264338213 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d000b870 con 0x7fe2dc15bed0 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2d9ffb700 1 -- 172.21.15.113:0/1264338213 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d000b9d0 con 0x7fe2dc15bed0 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2d9ffb700 1 -- 172.21.15.113:0/1264338213 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d000bd60 con 0x7fe2dc15bed0 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 -- 172.21.15.113:0/1264338213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc15bed0 msgr2=0x7fe2dc0b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 --2- 172.21.15.113:0/1264338213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0b19f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0d2a30 tx=0x7fe2d0009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 -- 172.21.15.113:0/1264338213 shutdown_connections 2024-01-16T13:52:15.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 --2- 172.21.15.113:0/1264338213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0b19f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 --2- 172.21.15.113:0/1264338213 >> v2:172.21.15.124:3300/0 conn(0x7fe2dc1603c0 0x7fe2dc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 -- 172.21.15.113:0/1264338213 >> 172.21.15.113:0/1264338213 conn(0x7fe2dc073130 msgr2=0x7fe2dc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:15.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 -- 172.21.15.113:0/1264338213 shutdown_connections 2024-01-16T13:52:15.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.944+0000 7fe2e173e700 1 -- 172.21.15.113:0/1264338213 wait complete. 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.946+0000 7fe2e173e700 1 Processor -- start 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.946+0000 7fe2e173e700 1 -- start start 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2e173e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc0d1a50 0x7fe2dc14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2e173e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2e173e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 0x7fe2dc0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2e173e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc14e660 con 0x7fe2dc1603c0 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2e173e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0ca680 con 0x7fe2dc15bed0 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2e173e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0c9b00 con 0x7fe2dc0d1a50 2024-01-16T13:52:15.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 0x7fe2dc0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 0x7fe2dc0cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48344/0 (socket says 172.21.15.113:48344) 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 -- 172.21.15.113:0/2188996524 learned_addr learned my addr 172.21.15.113:0/2188996524 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2da7fc700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2daffd700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc0d1a50 0x7fe2dc14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 -- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc0d1a50 msgr2=0x7fe2dc14e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc0d1a50 0x7fe2dc14e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 -- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc15bed0 msgr2=0x7fe2dc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.947+0000 7fe2db7fe700 1 -- 172.21.15.113:0/2188996524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d0009550 con 0x7fe2dc1603c0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2da7fc700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2db7fe700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 0x7fe2dc0cb2b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400cbd0 tx=0x7fe2d400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d400abb0 con 0x7fe2dc1603c0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d400ad10 con 0x7fe2dc1603c0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc0c9de0 con 0x7fe2dc1603c0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4007270 con 0x7fe2dc1603c0 2024-01-16T13:52:15.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.948+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc0ca3a0 con 0x7fe2dc1603c0 2024-01-16T13:52:15.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.949+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc0d0470 con 0x7fe2dc1603c0 2024-01-16T13:52:15.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.949+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe2d40073d0 con 0x7fe2dc1603c0 2024-01-16T13:52:15.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.949+0000 7fe2c37fe700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2c806b350 0x7fe2c806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:15.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.950+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2d4089990 con 0x7fe2dc1603c0 2024-01-16T13:52:15.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.950+0000 7fe2daffd700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2c806b350 0x7fe2c806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:15.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.950+0000 7fe2daffd700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2c806b350 0x7fe2c806d810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009130 tx=0x7fe2d0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:15.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:15.954+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2d4054830 con 0x7fe2dc1603c0 2024-01-16T13:52:16.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.186+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2dc067f60 con 0x7fe2dc1603c0 2024-01-16T13:52:16.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.186+0000 7fe2c37fe700 1 -- 172.21.15.113:0/2188996524 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fe2d4057e50 con 0x7fe2dc1603c0 2024-01-16T13:52:16.168 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:16.168 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2c806b350 msgr2=0x7fe2c806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2c806b350 0x7fe2c806d810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009130 tx=0x7fe2d0008030 comp rx=0 tx=0).stop 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 msgr2=0x7fe2dc0cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 0x7fe2dc0cb2b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400cbd0 tx=0x7fe2d400cf90 comp rx=0 tx=0).stop 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 shutdown_connections 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe2c806b350 0x7fe2c806d810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2dc1603c0 0x7fe2dc0cb2b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2dc0d1a50 0x7fe2dc14e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 --2- 172.21.15.113:0/2188996524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2dc15bed0 0x7fe2dc0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 >> 172.21.15.113:0/2188996524 conn(0x7fe2dc073130 msgr2=0x7fe2dc0b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:16.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 shutdown_connections 2024-01-16T13:52:16.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.188+0000 7fe2e173e700 1 -- 172.21.15.113:0/2188996524 wait complete. 2024-01-16T13:52:16.172 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:16.182 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:16.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.321+0000 7f545c7ec700 1 Processor -- start 2024-01-16T13:52:16.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.322+0000 7f545c7ec700 1 -- start start 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.322+0000 7f545c7ec700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f545415bed0 0x7f54540b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.322+0000 7f545c7ec700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54540d2160 0x7f54540d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545c7ec700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540d0770 con 0x7f54541603c0 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545c7ec700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545415c2b0 con 0x7f54540d2160 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545c7ec700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540cd100 con 0x7f545415bed0 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54540d2160 0x7f54540d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f54540d2160 0x7f54540d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48360/0 (socket says 172.21.15.113:48360) 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 -- 172.21.15.113:0/4037318446 learned_addr learned my addr 172.21.15.113:0/4037318446 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:16.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545a588700 1 --2- 172.21.15.113:0/4037318446 >> v2:172.21.15.124:3300/0 conn(0x7f545415bed0 0x7f54540b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 -- 172.21.15.113:0/4037318446 >> v2:172.21.15.124:3300/0 conn(0x7f545415bed0 msgr2=0x7f54540b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 --2- 172.21.15.113:0/4037318446 >> v2:172.21.15.124:3300/0 conn(0x7f545415bed0 0x7f54540b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 -- 172.21.15.113:0/4037318446 >> v1:172.21.15.124:6789/0 conn(0x7f54541603c0 legacy=0x7f54540d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 -- 172.21.15.113:0/4037318446 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540d09a0 con 0x7f54540d2160 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f545ad89700 1 --2- 172.21.15.113:0/4037318446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54540d2160 0x7f54540d0230 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f545409b2a0 tx=0x7f544c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7141ce236d853f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f5459586700 1 -- 172.21.15.113:0/4037318446 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00e070 con 0x7f54540d2160 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f5459586700 1 -- 172.21.15.113:0/4037318446 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c00b9c0 con 0x7f54540d2160 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.323+0000 7f5459586700 1 -- 172.21.15.113:0/4037318446 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00bd70 con 0x7f54540d2160 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 -- 172.21.15.113:0/4037318446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54540d2160 msgr2=0x7f54540d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 --2- 172.21.15.113:0/4037318446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54540d2160 0x7f54540d0230 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f545409b2a0 tx=0x7f544c00b330 comp rx=0 tx=0).stop 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 -- 172.21.15.113:0/4037318446 shutdown_connections 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 --2- 172.21.15.113:0/4037318446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54540d2160 0x7f54540d0230 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 --2- 172.21.15.113:0/4037318446 >> v2:172.21.15.124:3300/0 conn(0x7f545415bed0 0x7f54540b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 -- 172.21.15.113:0/4037318446 >> 172.21.15.113:0/4037318446 conn(0x7f5454073130 msgr2=0x7f545409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:16.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 -- 172.21.15.113:0/4037318446 shutdown_connections 2024-01-16T13:52:16.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.324+0000 7f545c7ec700 1 -- 172.21.15.113:0/4037318446 wait complete. 2024-01-16T13:52:16.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.325+0000 7f545c7ec700 1 Processor -- start 2024-01-16T13:52:16.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.325+0000 7f545c7ec700 1 -- start start 2024-01-16T13:52:16.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54540d2160 0x7f545409d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f545415bed0 0x7f5454156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 0x7f545415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540c5780 con 0x7f54541603c0 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540c5990 con 0x7f545415bed0 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540c5c70 con 0x7f54540d2160 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 0x7f545415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 0x7f545415e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48370/0 (socket says 172.21.15.113:48370) 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 -- 172.21.15.113:0/1846193898 learned_addr learned my addr 172.21.15.113:0/1846193898 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f5459d87700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f545415bed0 0x7f5454156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545a588700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54540d2160 0x7f545409d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 -- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54540d2160 msgr2=0x7f545409d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54540d2160 0x7f545409d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 -- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f545415bed0 msgr2=0x7f5454156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f545415bed0 0x7f5454156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 -- 172.21.15.113:0/1846193898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c009b30 con 0x7f54541603c0 2024-01-16T13:52:16.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545ad89700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 0x7f545415e5f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f54540cf7d0 tx=0x7f544c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c004880 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c0099b0 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54540c5f50 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.326+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c01c680 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.327+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545409d920 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.327+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5454067f60 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.328+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f544c01c7e0 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.328+0000 7f5446ffd700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f543806b350 0x7f543806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.328+0000 7f545a588700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f543806b350 0x7f543806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.328+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f544c08b420 con 0x7f54541603c0 2024-01-16T13:52:16.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.329+0000 7f545a588700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f543806b350 0x7f543806d810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54480099d0 tx=0x7f5448008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.333+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f544c056320 con 0x7f54541603c0 2024-01-16T13:52:16.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.525+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54540b7bd0 con 0x7f54541603c0 2024-01-16T13:52:16.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.527+0000 7f5446ffd700 1 -- 172.21.15.113:0/1846193898 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f544c059940 con 0x7f54541603c0 2024-01-16T13:52:16.510 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:16.510 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f543806b350 msgr2=0x7f543806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f543806b350 0x7f543806d810 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54480099d0 tx=0x7f5448008040 comp rx=0 tx=0).stop 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 msgr2=0x7f545415e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 0x7f545415e5f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f54540cf7d0 tx=0x7f544c004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 shutdown_connections 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f543806b350 0x7f543806d810 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54541603c0 0x7f545415e5f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54540d2160 0x7f545409d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 --2- 172.21.15.113:0/1846193898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f545415bed0 0x7f5454156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 >> 172.21.15.113:0/1846193898 conn(0x7f5454073130 msgr2=0x7f54540b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 shutdown_connections 2024-01-16T13:52:16.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.530+0000 7f545c7ec700 1 -- 172.21.15.113:0/1846193898 wait complete. 2024-01-16T13:52:16.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:16.524 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:16.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.666+0000 7f86d8bed700 1 Processor -- start 2024-01-16T13:52:16.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.666+0000 7f86d8bed700 1 -- start start 2024-01-16T13:52:16.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d8bed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 0x7f86d40b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d8bed700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f86d415f310 0x7f86d406fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d8bed700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d415ee80 con 0x7f86d40b3d10 2024-01-16T13:52:16.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d8bed700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d415e640 con 0x7f86d415f310 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d8bed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d40b5e90 con 0x7f86d40d0410 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d2d9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f86d40b3d10 0x7f86d4157e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39344/0 (socket says 172.21.15.113:39344) 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.667+0000 7f86d2d9d700 1 -- 172.21.15.113:0/2208072675 learned_addr learned my addr 172.21.15.113:0/2208072675 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d259c700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 0x7f86d40b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d1d9b700 1 --2- 172.21.15.113:0/2208072675 >> v2:172.21.15.124:3300/0 conn(0x7f86d415f310 0x7f86d406fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232850055 0 0) 0x7f86d415ee80 con 0x7f86d40b3d10 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc0035b0 con 0x7f86d40b3d10 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1998163383 0 0) 0x7f86bc0035b0 con 0x7f86d40b3d10 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 msgr2=0x7f86d40b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 0x7f86d40b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 >> v2:172.21.15.124:3300/0 conn(0x7f86d415f310 msgr2=0x7f86d406fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 --2- 172.21.15.113:0/2208072675 >> v2:172.21.15.124:3300/0 conn(0x7f86d415f310 0x7f86d406fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d415e830 con 0x7f86d40b3d10 2024-01-16T13:52:16.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f86cc0019f0 con 0x7f86d40b3d10 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 >> v1:172.21.15.113:6789/0 conn(0x7f86d40b3d10 legacy=0x7f86d4157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 0x7f86bc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86bc003f80 0x7f86bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d40b5e90 con 0x7f86bc004d90 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc00ba10 con 0x7f86bc003f80 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc00bba0 con 0x7f86d40d0410 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d1d9b700 1 --2- 172.21.15.113:0/2208072675 >> v2:172.21.15.124:3300/0 conn(0x7f86d415f310 0x7f86d406fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d259c700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 0x7f86bc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d1d9b700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86bc003f80 0x7f86bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 -- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 msgr2=0x7f86bc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d40d0410 0x7f86bc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 -- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86bc003f80 msgr2=0x7f86bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86bc003f80 0x7f86bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 -- 172.21.15.113:0/2208072675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc0035b0 con 0x7f86bc004d90 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.668+0000 7f86d2d9d700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86bc005210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00a860 tx=0x7f86bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283590300844ea6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc012600 con 0x7f86bc004d90 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc012760 con 0x7f86bc004d90 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d159a700 1 -- 172.21.15.113:0/2208072675 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc012b10 con 0x7f86bc004d90 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d8bed700 1 -- 172.21.15.113:0/2208072675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 msgr2=0x7f86bc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d8bed700 1 --2- 172.21.15.113:0/2208072675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86bc005210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00a860 tx=0x7f86bc005880 comp rx=0 tx=0).stop 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d2d9d700 1 -- 172.21.15.113:0/2208072675 reap_dead start 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d8bed700 1 -- 172.21.15.113:0/2208072675 shutdown_connections 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d8bed700 1 -- 172.21.15.113:0/2208072675 >> 172.21.15.113:0/2208072675 conn(0x7f86d40cd620 msgr2=0x7f86d40cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d8bed700 1 -- 172.21.15.113:0/2208072675 shutdown_connections 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.669+0000 7f86d8bed700 1 -- 172.21.15.113:0/2208072675 wait complete. 2024-01-16T13:52:16.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 Processor -- start 2024-01-16T13:52:16.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 -- start start 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86d4144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d4156b20 0x7f86d4159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86d4159a40 0x7f86d4147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4162420 con 0x7f86bc004d90 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d4148600 con 0x7f86d4159a40 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d8bed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d41456a0 con 0x7f86d4156b20 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86d4144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86d4144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48384/0 (socket says 172.21.15.113:48384) 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.671+0000 7f86d259c700 1 -- 172.21.15.113:0/3247453226 learned_addr learned my addr 172.21.15.113:0/3247453226 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d1d9b700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d4156b20 0x7f86d4159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d2d9d700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86d4159a40 0x7f86d4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d259c700 1 -- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d4156b20 msgr2=0x7f86d4159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d259c700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d4156b20 0x7f86d4159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d259c700 1 -- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86d4159a40 msgr2=0x7f86d4147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d259c700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86d4159a40 0x7f86d4147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d259c700 1 -- 172.21.15.113:0/3247453226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d4145940 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d2d9d700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86d4159a40 0x7f86d4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d259c700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86d4144f70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f86c80119a0 tx=0x7f86c8011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c800f940 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c800faa0 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d4145c20 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c800fe30 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.672+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d41461e0 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.673+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d4062a30 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.673+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f86c8014440 con 0x7f86bc004d90 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.674+0000 7f86c2ffd700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f86b406b350 0x7f86b406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.674+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86c808a1f0 con 0x7f86bc004d90 2024-01-16T13:52:16.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.674+0000 7f86d1d9b700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f86b406b350 0x7f86b406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.674+0000 7f86d1d9b700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f86b406b350 0x7f86b406d810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f86c4006f10 tx=0x7f86c4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.678+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86c8055060 con 0x7f86bc004d90 2024-01-16T13:52:16.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.868+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f86d4069020 con 0x7f86bc004d90 2024-01-16T13:52:16.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.868+0000 7f86c2ffd700 1 -- 172.21.15.113:0/3247453226 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86c8058680 con 0x7f86bc004d90 2024-01-16T13:52:16.851 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f86b406b350 msgr2=0x7f86b406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f86b406b350 0x7f86b406d810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f86c4006f10 tx=0x7f86c4008040 comp rx=0 tx=0).stop 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 msgr2=0x7f86d4144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86d4144f70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f86c80119a0 tx=0x7f86c8011cb0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 shutdown_connections 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f86b406b350 0x7f86b406d810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86d4159a40 0x7f86d4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86bc004d90 0x7f86d4144f70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 --2- 172.21.15.113:0/3247453226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f86d4156b20 0x7f86d4159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 >> 172.21.15.113:0/3247453226 conn(0x7f86d40cd620 msgr2=0x7f86d4073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 shutdown_connections 2024-01-16T13:52:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:16.870+0000 7f86d8bed700 1 -- 172.21.15.113:0/3247453226 wait complete. 2024-01-16T13:52:16.865 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:16.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.007+0000 7f93beb6d700 1 Processor -- start 2024-01-16T13:52:16.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.007+0000 7f93beb6d700 1 -- start start 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93beb6d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f93b815e280 0x7f93b80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93beb6d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93beb6d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f93b80b5050 0x7f93b80ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93beb6d700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b80b5500 con 0x7f93b815e280 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93beb6d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b80b42a0 con 0x7f93b80b5050 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93beb6d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b80af900 con 0x7f93b8073f90 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f93b815e280 0x7f93b80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f93b815e280 0x7f93b80b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48396/0 (socket says 172.21.15.113:48396) 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 -- 172.21.15.113:0/1777305678 learned_addr learned my addr 172.21.15.113:0/1777305678 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bd10a700 1 --2- 172.21.15.113:0/1777305678 >> v2:172.21.15.124:3300/0 conn(0x7f93b80b5050 0x7f93b80ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93b7fff700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 -- 172.21.15.113:0/1777305678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 msgr2=0x7f93b80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b80b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 -- 172.21.15.113:0/1777305678 >> v2:172.21.15.124:3300/0 conn(0x7f93b80b5050 msgr2=0x7f93b80ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 --2- 172.21.15.113:0/1777305678 >> v2:172.21.15.124:3300/0 conn(0x7f93b80b5050 0x7f93b80ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 -- 172.21.15.113:0/1777305678 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8074d30 con 0x7f93b815e280 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93b7fff700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93bc909700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815e280 0x7f93b80b3d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f93b80731f0 tx=0x7f93ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284f212a24d8c0c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93b77fe700 1 -- 172.21.15.113:0/1777305678 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac011830 con 0x7f93b815e280 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93b77fe700 1 -- 172.21.15.113:0/1777305678 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac011990 con 0x7f93b815e280 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.008+0000 7f93b77fe700 1 -- 172.21.15.113:0/1777305678 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac011d20 con 0x7f93b815e280 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 -- 172.21.15.113:0/1777305678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815e280 msgr2=0x7f93b80b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815e280 0x7f93b80b3d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f93b80731f0 tx=0x7f93ac009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 -- 172.21.15.113:0/1777305678 shutdown_connections 2024-01-16T13:52:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b815e280 0x7f93b80b3d60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1777305678 >> v2:172.21.15.124:3300/0 conn(0x7f93b80b5050 0x7f93b80ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1777305678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 -- 172.21.15.113:0/1777305678 >> 172.21.15.113:0/1777305678 conn(0x7f93b8072b20 msgr2=0x7f93b806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:16.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 -- 172.21.15.113:0/1777305678 shutdown_connections 2024-01-16T13:52:16.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.009+0000 7f93beb6d700 1 -- 172.21.15.113:0/1777305678 wait complete. 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.010+0000 7f93beb6d700 1 Processor -- start 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.010+0000 7f93beb6d700 1 -- start start 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93beb6d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b8155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93beb6d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93b80b5050 0x7f93b8146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93beb6d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 0x7f93b8153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93beb6d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b815eff0 con 0x7f93b81527c0 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93beb6d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b8155bc0 con 0x7f93b80b5050 2024-01-16T13:52:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93beb6d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b8154270 con 0x7f93b8073f90 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 0x7f93b8153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 0x7f93b8153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48402/0 (socket says 172.21.15.113:48402) 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 -- 172.21.15.113:0/1902192629 learned_addr learned my addr 172.21.15.113:0/1902192629 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bc909700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b8155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93b7fff700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93b80b5050 0x7f93b8146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 -- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 msgr2=0x7f93b8155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b8155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 -- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93b80b5050 msgr2=0x7f93b8146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93b80b5050 0x7f93b8146f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93bd10a700 1 -- 172.21.15.113:0/1902192629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac009550 con 0x7f93b81527c0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.011+0000 7f93b7fff700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93b80b5050 0x7f93b8146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.012+0000 7f93bd10a700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 0x7f93b8153620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d30 tx=0x7f93b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.012+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b000cd40 con 0x7f93b81527c0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.012+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b000f040 con 0x7f93b81527c0 2024-01-16T13:52:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.012+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b00146c0 con 0x7f93b81527c0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.012+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b815e6e0 con 0x7f93b81527c0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.012+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b815e930 con 0x7f93b81527c0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.013+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b815f130 con 0x7f93b81527c0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.013+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f93b0014820 con 0x7f93b81527c0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.013+0000 7f93b57fa700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f939c06b350 0x7f939c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.014+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93b008a2d0 con 0x7f93b81527c0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.014+0000 7f93bc909700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f939c06b350 0x7f939c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:16.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.014+0000 7f93bc909700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f939c06b350 0x7f939c06d810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009c10 tx=0x7f93ac008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.018+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93b815f130 con 0x7f93b81527c0 2024-01-16T13:52:17.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.207+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93b815f130 con 0x7f93b81527c0 2024-01-16T13:52:17.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.207+0000 7f93b57fa700 1 -- 172.21.15.113:0/1902192629 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f93b815f130 con 0x7f93b81527c0 2024-01-16T13:52:17.190 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f939c06b350 msgr2=0x7f939c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f939c06b350 0x7f939c06d810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009c10 tx=0x7f93ac008030 comp rx=0 tx=0).stop 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 msgr2=0x7f93b8153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 0x7f93b8153620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d30 tx=0x7f93b000c3a0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 shutdown_connections 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f939c06b350 0x7f939c06d810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93b81527c0 0x7f93b8153620 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93b8073f90 0x7f93b8155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 --2- 172.21.15.113:0/1902192629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93b80b5050 0x7f93b8146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.209+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 >> 172.21.15.113:0/1902192629 conn(0x7f93b8072b20 msgr2=0x7f93b80afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.210+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 shutdown_connections 2024-01-16T13:52:17.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.210+0000 7f93beb6d700 1 -- 172.21.15.113:0/1902192629 wait complete. 2024-01-16T13:52:17.204 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:17.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.349+0000 7fae4ef27700 1 Processor -- start 2024-01-16T13:52:17.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.350+0000 7fae4ef27700 1 -- start start 2024-01-16T13:52:17.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.350+0000 7fae4ef27700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fae4815e280 0x7fae480b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.350+0000 7fae4ef27700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 0x7fae480b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.350+0000 7fae4ef27700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae480b42a0 con 0x7fae480b5050 2024-01-16T13:52:17.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.350+0000 7fae4ef27700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae480b5500 con 0x7fae4815e280 2024-01-16T13:52:17.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.350+0000 7fae4ef27700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae480af900 con 0x7fae48073f90 2024-01-16T13:52:17.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 0x7fae480b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae4ccc3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fae4815e280 0x7fae480b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 0x7fae480b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:44382/0 (socket says 172.21.15.113:44382) 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 -- 172.21.15.113:0/1338898559 learned_addr learned my addr 172.21.15.113:0/1338898559 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 -- 172.21.15.113:0/1338898559 >> v2:172.21.15.124:3300/0 conn(0x7fae4815e280 msgr2=0x7fae480b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 --2- 172.21.15.113:0/1338898559 >> v2:172.21.15.124:3300/0 conn(0x7fae4815e280 0x7fae480b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 -- 172.21.15.113:0/1338898559 >> v1:172.21.15.124:6789/0 conn(0x7fae480b5050 legacy=0x7fae480ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 -- 172.21.15.113:0/1338898559 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae48074d30 con 0x7fae48073f90 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae4ccc3700 1 --2- 172.21.15.113:0/1338898559 >> v2:172.21.15.124:3300/0 conn(0x7fae4815e280 0x7fae480b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae47fff700 1 --2- 172.21.15.113:0/1338898559 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 0x7fae480b4b10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fae340098f0 tx=0x7fae34009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c79ab31bdd0927 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae477fe700 1 -- 172.21.15.113:0/1338898559 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34011830 con 0x7fae48073f90 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae477fe700 1 -- 172.21.15.113:0/1338898559 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae34011990 con 0x7fae48073f90 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.351+0000 7fae477fe700 1 -- 172.21.15.113:0/1338898559 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34011d20 con 0x7fae48073f90 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 -- 172.21.15.113:0/1338898559 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 msgr2=0x7fae480b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1338898559 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 0x7fae480b4b10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fae340098f0 tx=0x7fae34009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:17.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 -- 172.21.15.113:0/1338898559 shutdown_connections 2024-01-16T13:52:17.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1338898559 >> v2:172.21.15.124:3300/0 conn(0x7fae4815e280 0x7fae480b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1338898559 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae48073f90 0x7fae480b4b10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 -- 172.21.15.113:0/1338898559 >> 172.21.15.113:0/1338898559 conn(0x7fae48072b20 msgr2=0x7fae4806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:17.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 -- 172.21.15.113:0/1338898559 shutdown_connections 2024-01-16T13:52:17.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.352+0000 7fae4ef27700 1 -- 172.21.15.113:0/1338898559 wait complete. 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.353+0000 7fae4ef27700 1 Processor -- start 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.353+0000 7fae4ef27700 1 -- start start 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ef27700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae48073f90 0x7fae481505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ef27700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 0x7fae480b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ef27700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae4815e6e0 0x7fae480b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ef27700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae481f7930 con 0x7fae480b5050 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ef27700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae481f6cb0 con 0x7fae48073f90 2024-01-16T13:52:17.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ef27700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae481f6f90 con 0x7fae4815e6e0 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 0x7fae480b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 0x7fae480b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48408/0 (socket says 172.21.15.113:48408) 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 -- 172.21.15.113:0/1836662342 learned_addr learned my addr 172.21.15.113:0/1836662342 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4ccc3700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae48073f90 0x7fae481505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae4d4c4700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae4815e6e0 0x7fae480b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 -- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae4815e6e0 msgr2=0x7fae480b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae4815e6e0 0x7fae480b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 -- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae48073f90 msgr2=0x7fae481505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae48073f90 0x7fae481505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.354+0000 7fae47fff700 1 -- 172.21.15.113:0/1836662342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0094a0 con 0x7fae480b5050 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae4ccc3700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae48073f90 0x7fae481505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae4d4c4700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae4815e6e0 0x7fae480b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:17.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae47fff700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 0x7fae480b19b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae34009c40 tx=0x7fae34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae340036a0 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3401e070 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae34003a60 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae34009550 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.355+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae481f7550 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.356+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae48074750 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.356+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fae34003bc0 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.356+0000 7fae457fa700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fae2c06b350 0x7fae2c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.357+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae3408aa90 con 0x7fae480b5050 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.357+0000 7fae4ccc3700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fae2c06b350 0x7fae2c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.357+0000 7fae4ccc3700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fae2c06b350 0x7fae2c06d810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae48155cf0 tx=0x7fae3c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.361+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae48074750 con 0x7fae480b5050 2024-01-16T13:52:17.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.553+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae481f7a70 con 0x7fae480b5050 2024-01-16T13:52:17.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.553+0000 7fae457fa700 1 -- 172.21.15.113:0/1836662342 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae481f7a70 con 0x7fae480b5050 2024-01-16T13:52:17.535 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:17.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fae2c06b350 msgr2=0x7fae2c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fae2c06b350 0x7fae2c06d810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae48155cf0 tx=0x7fae3c008040 comp rx=0 tx=0).stop 2024-01-16T13:52:17.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 msgr2=0x7fae480b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 0x7fae480b19b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae34009c40 tx=0x7fae34004750 comp rx=0 tx=0).stop 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 shutdown_connections 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fae2c06b350 0x7fae2c06d810 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae4815e6e0 0x7fae480b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae48073f90 0x7fae481505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 --2- 172.21.15.113:0/1836662342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae480b5050 0x7fae480b19b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 >> 172.21.15.113:0/1836662342 conn(0x7fae48072b20 msgr2=0x7fae4815d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 shutdown_connections 2024-01-16T13:52:17.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.555+0000 7fae4ef27700 1 -- 172.21.15.113:0/1836662342 wait complete. 2024-01-16T13:52:17.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:17.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.688+0000 7f2e20720700 1 Processor -- start 2024-01-16T13:52:17.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.688+0000 7f2e20720700 1 -- start start 2024-01-16T13:52:17.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.689+0000 7f2e20720700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2e1815ac50 0x7f2e1806c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.689+0000 7f2e20720700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1806bf50 con 0x7f2e180b50c0 2024-01-16T13:52:17.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.689+0000 7f2e20720700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1815b030 con 0x7f2e1806f460 2024-01-16T13:52:17.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.689+0000 7f2e20720700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1806e810 con 0x7f2e1815ac50 2024-01-16T13:52:17.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.689+0000 7f2e1ecbd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2e180b50c0 0x7f2e180cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39358/0 (socket says 172.21.15.113:39358) 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.689+0000 7f2e1ecbd700 1 -- 172.21.15.113:0/338421615 learned_addr learned my addr 172.21.15.113:0/338421615 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1e4bc700 1 --2- 172.21.15.113:0/338421615 >> v2:172.21.15.124:3300/0 conn(0x7f2e1815ac50 0x7f2e1806c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385242368 0 0) 0x7f2e1806bf50 con 0x7f2e180b50c0 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e040035b0 con 0x7f2e180b50c0 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830683715 0 0) 0x7f2e1815b030 con 0x7f2e1806f460 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e1806bf50 con 0x7f2e1806f460 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1e4bc700 1 -- 172.21.15.113:0/338421615 >> v1:172.21.15.124:6789/0 conn(0x7f2e1806f460 legacy=0x7f2e180b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1e4bc700 1 -- 172.21.15.113:0/338421615 >> v1:172.21.15.113:6789/0 conn(0x7f2e180b50c0 legacy=0x7f2e180cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432582271 0 0) 0x7f2e040035b0 con 0x7f2e180b50c0 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1e4bc700 1 -- 172.21.15.113:0/338421615 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1806b340 con 0x7f2e1815ac50 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1e4bc700 1 --2- 172.21.15.113:0/338421615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e1806c3e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009840 tx=0x7f2e08009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc34fd4624ef6a30 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e100019f0 con 0x7f2e1815ac50 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08004030 con 0x7f2e1815ac50 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.690+0000 7f2e1d4ba700 1 -- 172.21.15.113:0/338421615 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080043c0 con 0x7f2e1815ac50 2024-01-16T13:52:17.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 -- 172.21.15.113:0/338421615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 msgr2=0x7f2e1806c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 --2- 172.21.15.113:0/338421615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e1806c3e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009840 tx=0x7f2e08009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:17.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 -- 172.21.15.113:0/338421615 shutdown_connections 2024-01-16T13:52:17.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 --2- 172.21.15.113:0/338421615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e1806c3e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 -- 172.21.15.113:0/338421615 >> 172.21.15.113:0/338421615 conn(0x7f2e180b3130 msgr2=0x7f2e180d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:17.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 -- 172.21.15.113:0/338421615 shutdown_connections 2024-01-16T13:52:17.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.691+0000 7f2e20720700 1 -- 172.21.15.113:0/338421615 wait complete. 2024-01-16T13:52:17.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.692+0000 7f2e20720700 1 Processor -- start 2024-01-16T13:52:17.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.692+0000 7f2e20720700 1 -- start start 2024-01-16T13:52:17.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2e1806f460 0x7f2e1814b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 0x7f2e181505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e18151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18150f40 con 0x7f2e180b50c0 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18151220 con 0x7f2e1815ac50 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18151500 con 0x7f2e1806f460 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 0x7f2e181505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 0x7f2e181505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48412/0 (socket says 172.21.15.113:48412) 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 -- 172.21.15.113:0/4045398065 learned_addr learned my addr 172.21.15.113:0/4045398065 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1e4bc700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2e1806f460 0x7f2e1814b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1ecbd700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e18151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 -- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2e1806f460 msgr2=0x7f2e1814b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2e1806f460 0x7f2e1814b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 -- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 msgr2=0x7f2e18151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e18151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 -- 172.21.15.113:0/4045398065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080094a0 con 0x7f2e180b50c0 2024-01-16T13:52:17.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1e4bc700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2e1806f460 0x7f2e1814b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1ecbd700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e18151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e1dcbb700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 0x7f2e181505f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c500 tx=0x7f2e1400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e140052d0 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e14005450 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.693+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e181517e0 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.694+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e140038c0 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.694+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1814bf00 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.694+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18067f60 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.695+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2e14003a20 con 0x7f2e180b50c0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.695+0000 7f2e0effd700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2dfc06b350 0x7f2dfc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.696+0000 7f2e1e4bc700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2dfc06b350 0x7f2dfc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:17.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.696+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e14089110 con 0x7f2e180b50c0 2024-01-16T13:52:17.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.696+0000 7f2e1e4bc700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2dfc06b350 0x7f2dfc06d810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e080090b0 tx=0x7f2e08008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:17.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.700+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e14053ef0 con 0x7f2e180b50c0 2024-01-16T13:52:17.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.932+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e1806e350 con 0x7f2e180b50c0 2024-01-16T13:52:17.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.933+0000 7f2e0effd700 1 -- 172.21.15.113:0/4045398065 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f2e14057510 con 0x7f2e180b50c0 2024-01-16T13:52:17.915 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:17.915 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:17.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2dfc06b350 msgr2=0x7f2dfc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2dfc06b350 0x7f2dfc06d810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e080090b0 tx=0x7f2e08008030 comp rx=0 tx=0).stop 2024-01-16T13:52:17.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 msgr2=0x7f2e181505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 0x7f2e181505f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c500 tx=0x7f2e1400c8c0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 shutdown_connections 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2dfc06b350 0x7f2dfc06d810 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2e1815ac50 0x7f2e18151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2e1806f460 0x7f2e1814b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 --2- 172.21.15.113:0/4045398065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180b50c0 0x7f2e181505f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 >> 172.21.15.113:0/4045398065 conn(0x7f2e180b3130 msgr2=0x7f2e180cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 shutdown_connections 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:17.935+0000 7f2e20720700 1 -- 172.21.15.113:0/4045398065 wait complete. 2024-01-16T13:52:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:17.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:18.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 Processor -- start 2024-01-16T13:52:18.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 -- start start 2024-01-16T13:52:18.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 0x7f19c015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f19c0062310 0x7f19c00d0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c015fa50 con 0x7f19c015d0f0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c015fca0 con 0x7f19c0062310 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.068+0000 7f19c703a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015ff80 con 0x7f19c015c7d0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c55d7700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f19c015d0f0 0x7f19c00726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39362/0 (socket says 172.21.15.113:39362) 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c55d7700 1 -- 172.21.15.113:0/2262501726 learned_addr learned my addr 172.21.15.113:0/2262501726 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 0x7f19c015cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727143022 0 0) 0x7f19c015fa50 con 0x7f19c015d0f0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bffff700 1 --2- 172.21.15.113:0/2262501726 >> v2:172.21.15.124:3300/0 conn(0x7f19c0062310 0x7f19c00d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b00035b0 con 0x7f19c015d0f0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 575127544 0 0) 0x7f19b00035b0 con 0x7f19c015d0f0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 msgr2=0x7f19c015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 0x7f19c015cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 >> v2:172.21.15.124:3300/0 conn(0x7f19c0062310 msgr2=0x7f19c00d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 --2- 172.21.15.113:0/2262501726 >> v2:172.21.15.124:3300/0 conn(0x7f19c0062310 0x7f19c00d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0160260 con 0x7f19c015d0f0 2024-01-16T13:52:18.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f19b80019f0 con 0x7f19c015d0f0 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 >> v1:172.21.15.113:6789/0 conn(0x7f19c015d0f0 legacy=0x7f19c00726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0003a00 0x7f19b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 0x7f19b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0004d90 0x7f19b0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015fca0 con 0x7f19b0003a00 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015ff80 con 0x7f19b0004d90 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b000ba10 con 0x7f19c015c7d0 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0003a00 0x7f19b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c55d7700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0004d90 0x7f19b0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19bffff700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 0x7f19b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 -- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 msgr2=0x7f19b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c015c7d0 0x7f19b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 -- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0004d90 msgr2=0x7f19b0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:18.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0004d90 0x7f19b0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.069+0000 7f19c4dd6700 1 -- 172.21.15.113:0/2262501726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b00035b0 con 0x7f19b0003a00 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0003a00 0x7f19b0003e20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009b60 tx=0x7f19b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc3d2794551d270 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b400e6c0 con 0x7f19b0003a00 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b400e820 con 0x7f19b0003a00 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19bf7fe700 1 -- 172.21.15.113:0/2262501726 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b400eb90 con 0x7f19b0003a00 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c703a700 1 -- 172.21.15.113:0/2262501726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0003a00 msgr2=0x7f19b0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c703a700 1 --2- 172.21.15.113:0/2262501726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19b0003a00 0x7f19b0003e20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009b60 tx=0x7f19b0005880 comp rx=0 tx=0).stop 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c55d7700 1 -- 172.21.15.113:0/2262501726 reap_dead start 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c703a700 1 -- 172.21.15.113:0/2262501726 shutdown_connections 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c703a700 1 -- 172.21.15.113:0/2262501726 >> 172.21.15.113:0/2262501726 conn(0x7f19c00b5640 msgr2=0x7f19c00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c703a700 1 -- 172.21.15.113:0/2262501726 shutdown_connections 2024-01-16T13:52:18.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.070+0000 7f19c703a700 1 -- 172.21.15.113:0/2262501726 wait complete. 2024-01-16T13:52:18.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 Processor -- start 2024-01-16T13:52:18.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 -- start start 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0003a00 0x7f19c014f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 0x7f19c01509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c01f3470 0x7f19c0153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00b5260 con 0x7f19c00b4de0 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0150f00 con 0x7f19b0003a00 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.072+0000 7f19c703a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c014f920 con 0x7f19c01f3470 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 0x7f19c01509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 0x7f19c01509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48422/0 (socket says 172.21.15.113:48422) 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 -- 172.21.15.113:0/803658872 learned_addr learned my addr 172.21.15.113:0/803658872 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0003a00 0x7f19c014f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19c55d7700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c01f3470 0x7f19c0153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 -- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c01f3470 msgr2=0x7f19c0153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c01f3470 0x7f19c0153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 -- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0003a00 msgr2=0x7f19c014f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0003a00 0x7f19c014f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 -- 172.21.15.113:0/803658872 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c014fc00 con 0x7f19c00b4de0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bffff700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 0x7f19c01509c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f19ac011a30 tx=0x7f19ac011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac004240 con 0x7f19c00b4de0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac0043a0 con 0x7f19c00b4de0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c014fee0 con 0x7f19c00b4de0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00f7f0 con 0x7f19c00b4de0 2024-01-16T13:52:18.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.073+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c01504a0 con 0x7f19c00b4de0 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.074+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c0067f60 con 0x7f19c00b4de0 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.074+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f19ac00f950 con 0x7f19c00b4de0 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.075+0000 7f19bd7fa700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f19a406b350 0x7f19a406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.075+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19ac08a1a0 con 0x7f19c00b4de0 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.075+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f19a406b350 0x7f19a406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.075+0000 7f19c4dd6700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f19a406b350 0x7f19a406d810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f19b40090b0 tx=0x7f19b4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:18.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.079+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19ac054f80 con 0x7f19c00b4de0 2024-01-16T13:52:18.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.271+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19c00cc100 con 0x7f19c00b4de0 2024-01-16T13:52:18.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.272+0000 7f19bd7fa700 1 -- 172.21.15.113:0/803658872 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f19ac019090 con 0x7f19c00b4de0 2024-01-16T13:52:18.254 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:18.255 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f19a406b350 msgr2=0x7f19a406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f19a406b350 0x7f19a406d810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f19b40090b0 tx=0x7f19b4008030 comp rx=0 tx=0).stop 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 msgr2=0x7f19c01509c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 0x7f19c01509c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f19ac011a30 tx=0x7f19ac011df0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 shutdown_connections 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f19a406b350 0x7f19a406d810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19c01f3470 0x7f19c0153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19b0003a00 0x7f19c014f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 --2- 172.21.15.113:0/803658872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19c00b4de0 0x7f19c01509c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 >> 172.21.15.113:0/803658872 conn(0x7f19c00b5640 msgr2=0x7f19c00ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:18.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.274+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 shutdown_connections 2024-01-16T13:52:18.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.275+0000 7f19c703a700 1 -- 172.21.15.113:0/803658872 wait complete. 2024-01-16T13:52:18.269 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:18.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.408+0000 7fac03ec8700 1 Processor -- start 2024-01-16T13:52:18.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.408+0000 7fac03ec8700 1 -- start start 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac03ec8700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabfc15af10 0x7fabfc15e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac03ec8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc0ce140 0x7fabfc15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac03ec8700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc1414c0 con 0x7fabfc15f460 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac03ec8700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc15f8c0 con 0x7fabfc15af10 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac03ec8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0ce9f0 con 0x7fabfc0ce140 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabfc15af10 0x7fabfc15e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabfc15af10 0x7fabfc15e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48424/0 (socket says 172.21.15.113:48424) 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 -- 172.21.15.113:0/4164113098 learned_addr learned my addr 172.21.15.113:0/4164113098 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac02465700 1 --1- 172.21.15.113:0/4164113098 >> v1:172.21.15.113:6789/0 conn(0x7fabfc15f460 0x7fabfc15b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:39368/0 (socket says 172.21.15.113:39368) 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01463700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc0ce140 0x7fabfc15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 -- 172.21.15.113:0/4164113098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc0ce140 msgr2=0x7fabfc15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:18.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc0ce140 0x7fabfc15ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 -- 172.21.15.113:0/4164113098 >> v1:172.21.15.113:6789/0 conn(0x7fabfc15f460 legacy=0x7fabfc15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 -- 172.21.15.113:0/4164113098 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0cecd0 con 0x7fabfc15af10 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac00c62700 1 -- 172.21.15.113:0/4164113098 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651551043 0 0) 0x7fabfc1414c0 con 0x7fabfc15f460 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01463700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc0ce140 0x7fabfc15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac01c64700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc15af10 0x7fabfc15e9e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098f0 tx=0x7fabf8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68ba4ee59c6d2f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac00c62700 1 -- 172.21.15.113:0/4164113098 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf800b8a0 con 0x7fabfc15af10 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac00c62700 1 -- 172.21.15.113:0/4164113098 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf800ba00 con 0x7fabfc15af10 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.409+0000 7fac00c62700 1 -- 172.21.15.113:0/4164113098 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf800bd90 con 0x7fabfc15af10 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 -- 172.21.15.113:0/4164113098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc15af10 msgr2=0x7fabfc15e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc15af10 0x7fabfc15e9e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098f0 tx=0x7fabf8009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 -- 172.21.15.113:0/4164113098 shutdown_connections 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc15af10 0x7fabfc15e9e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 --2- 172.21.15.113:0/4164113098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc0ce140 0x7fabfc15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 -- 172.21.15.113:0/4164113098 >> 172.21.15.113:0/4164113098 conn(0x7fabfc0cd570 msgr2=0x7fabfc0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:18.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 -- 172.21.15.113:0/4164113098 shutdown_connections 2024-01-16T13:52:18.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.410+0000 7fac03ec8700 1 -- 172.21.15.113:0/4164113098 wait complete. 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.411+0000 7fac03ec8700 1 Processor -- start 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.411+0000 7fac03ec8700 1 -- start start 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac03ec8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 0x7fabfc159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac03ec8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc15af10 0x7fabfc0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac03ec8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fabfc15f460 0x7fabfc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac03ec8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc13e580 con 0x7fabfc0ce140 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac03ec8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc165ec0 con 0x7fabfc15f460 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac03ec8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc1661a0 con 0x7fabfc15af10 2024-01-16T13:52:18.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 0x7fabfc159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 0x7fabfc159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48440/0 (socket says 172.21.15.113:48440) 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 -- 172.21.15.113:0/1921971592 learned_addr learned my addr 172.21.15.113:0/1921971592 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01463700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc15af10 0x7fabfc0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac02465700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fabfc15f460 0x7fabfc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 -- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc15af10 msgr2=0x7fabfc0b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc15af10 0x7fabfc0b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 -- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fabfc15f460 msgr2=0x7fabfc13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fabfc15f460 0x7fabfc13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 -- 172.21.15.113:0/1921971592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8009550 con 0x7fabfc0ce140 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac02465700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fabfc15f460 0x7fabfc13c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.412+0000 7fac01c64700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 0x7fabfc159800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009c40 tx=0x7fabf8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.413+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf80036a0 con 0x7fabfc0ce140 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.413+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf801f070 con 0x7fabfc0ce140 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.413+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc166480 con 0x7fabfc0ce140 2024-01-16T13:52:18.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.413+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8003a60 con 0x7fabfc0ce140 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.413+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc159d40 con 0x7fabfc0ce140 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.414+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc159ed0 con 0x7fabfc0ce140 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.414+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fabf8003bc0 con 0x7fabfc0ce140 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.415+0000 7fabf27fc700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fabe006b380 0x7fabe006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.415+0000 7fac01463700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fabe006b380 0x7fabe006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.415+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabf8021070 con 0x7fabfc0ce140 2024-01-16T13:52:18.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.415+0000 7fac01463700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fabe006b380 0x7fabe006d840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b5b30 tx=0x7fabec008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:18.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.419+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fabf80563e0 con 0x7fabfc0ce140 2024-01-16T13:52:18.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.654+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabfc159ed0 con 0x7fabfc0ce140 2024-01-16T13:52:18.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.655+0000 7fabf27fc700 1 -- 172.21.15.113:0/1921971592 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fabfc159ed0 con 0x7fabfc0ce140 2024-01-16T13:52:18.637 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:18.637 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fabe006b380 msgr2=0x7fabe006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fabe006b380 0x7fabe006d840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b5b30 tx=0x7fabec008040 comp rx=0 tx=0).stop 2024-01-16T13:52:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 msgr2=0x7fabfc159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 0x7fabfc159800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009c40 tx=0x7fabf8004750 comp rx=0 tx=0).stop 2024-01-16T13:52:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 shutdown_connections 2024-01-16T13:52:18.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fabe006b380 0x7fabe006d840 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fabfc15f460 0x7fabfc13c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabfc0ce140 0x7fabfc159800 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 --2- 172.21.15.113:0/1921971592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fabfc15af10 0x7fabfc0b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 >> 172.21.15.113:0/1921971592 conn(0x7fabfc0cd570 msgr2=0x7fabfc06dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 shutdown_connections 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:18.657+0000 7fac03ec8700 1 -- 172.21.15.113:0/1921971592 wait complete. 2024-01-16T13:52:18.640 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:19.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T13:52:04.292443+0000', 'modified': '2024-01-16T13:52:07.316433+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.124:6833/4249021810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.124:6832', 'nonce': 4249021810}, {'type': 'v1', 'addr': '172.21.15.124:6833', 'nonce': 4249021810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T13:52:19.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T13:52:19.054 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-16T13:52:19.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.200+0000 7f4f239ac700 1 Processor -- start 2024-01-16T13:52:19.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.200+0000 7f4f239ac700 1 -- start start 2024-01-16T13:52:19.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f239ac700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4f1c15e2a0 0x7f4f1c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f239ac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c0ce090 0x7f4f1c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f239ac700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4f1c073f90 0x7f4f1c074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f239ac700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c074960 con 0x7f4f1c15e2a0 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f239ac700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c06e7f0 con 0x7f4f1c073f90 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f239ac700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c06dec0 con 0x7f4f1c0ce090 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4f1c15e2a0 0x7f4f1c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4f1c15e2a0 0x7f4f1c06ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48450/0 (socket says 172.21.15.113:48450) 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 -- 172.21.15.113:0/2692867038 learned_addr learned my addr 172.21.15.113:0/2692867038 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21f49700 1 --2- 172.21.15.113:0/2692867038 >> v2:172.21.15.124:3300/0 conn(0x7f4f1c073f90 0x7f4f1c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f20f47700 1 --2- 172.21.15.113:0/2692867038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c0ce090 0x7f4f1c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 -- 172.21.15.113:0/2692867038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c0ce090 msgr2=0x7f4f1c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 --2- 172.21.15.113:0/2692867038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c0ce090 0x7f4f1c06ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 -- 172.21.15.113:0/2692867038 >> v2:172.21.15.124:3300/0 conn(0x7f4f1c073f90 msgr2=0x7f4f1c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 --2- 172.21.15.113:0/2692867038 >> v2:172.21.15.124:3300/0 conn(0x7f4f1c073f90 0x7f4f1c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 -- 172.21.15.113:0/2692867038 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c06e1a0 con 0x7f4f1c15e2a0 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f21748700 1 --2- 172.21.15.113:0/2692867038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c15e2a0 0x7f4f1c06ba70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c06c3c0 tx=0x7f4f18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9937ef35e9963051 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f13fff700 1 -- 172.21.15.113:0/2692867038 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1800b870 con 0x7f4f1c15e2a0 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f13fff700 1 -- 172.21.15.113:0/2692867038 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1800b9d0 con 0x7f4f1c15e2a0 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.201+0000 7f4f13fff700 1 -- 172.21.15.113:0/2692867038 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1800bd60 con 0x7f4f1c15e2a0 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 -- 172.21.15.113:0/2692867038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c15e2a0 msgr2=0x7f4f1c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 --2- 172.21.15.113:0/2692867038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c15e2a0 0x7f4f1c06ba70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c06c3c0 tx=0x7f4f18009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 -- 172.21.15.113:0/2692867038 shutdown_connections 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 --2- 172.21.15.113:0/2692867038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c15e2a0 0x7f4f1c06ba70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 --2- 172.21.15.113:0/2692867038 >> v2:172.21.15.124:3300/0 conn(0x7f4f1c073f90 0x7f4f1c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 --2- 172.21.15.113:0/2692867038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c0ce090 0x7f4f1c06ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 -- 172.21.15.113:0/2692867038 >> 172.21.15.113:0/2692867038 conn(0x7f4f1c0cd570 msgr2=0x7f4f1c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 -- 172.21.15.113:0/2692867038 shutdown_connections 2024-01-16T13:52:19.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.202+0000 7f4f239ac700 1 -- 172.21.15.113:0/2692867038 wait complete. 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.203+0000 7f4f239ac700 1 Processor -- start 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.203+0000 7f4f239ac700 1 -- start start 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f239ac700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 0x7f4f1c158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f239ac700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f1c0ce090 0x7f4f1c15ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f239ac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c15e2a0 0x7f4f1c146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f239ac700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c158dc0 con 0x7f4f1c073f90 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f239ac700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c15f2a0 con 0x7f4f1c0ce090 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f239ac700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0ad870 con 0x7f4f1c15e2a0 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 0x7f4f1c158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 0x7f4f1c158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48456/0 (socket says 172.21.15.113:48456) 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 -- 172.21.15.113:0/687885320 learned_addr learned my addr 172.21.15.113:0/687885320 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21f49700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c15e2a0 0x7f4f1c146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f20f47700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f1c0ce090 0x7f4f1c15ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 -- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c15e2a0 msgr2=0x7f4f1c146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c15e2a0 0x7f4f1c146490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 -- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f1c0ce090 msgr2=0x7f4f1c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f1c0ce090 0x7f4f1c15ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21748700 1 -- 172.21.15.113:0/687885320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f18009550 con 0x7f4f1c073f90 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f21f49700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c15e2a0 0x7f4f1c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.204+0000 7f4f20f47700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f1c0ce090 0x7f4f1c15ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.205+0000 7f4f21748700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 0x7f4f1c158880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4f18000c00 tx=0x7f4f18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.205+0000 7f4f11ffb700 1 -- 172.21.15.113:0/687885320 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18004540 con 0x7f4f1c073f90 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.205+0000 7f4f11ffb700 1 -- 172.21.15.113:0/687885320 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f180036a0 con 0x7f4f1c073f90 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.205+0000 7f4f11ffb700 1 -- 172.21.15.113:0/687885320 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18003a50 con 0x7f4f1c073f90 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.205+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c1469d0 con 0x7f4f1c073f90 2024-01-16T13:52:19.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.205+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c0adac0 con 0x7f4f1c073f90 2024-01-16T13:52:19.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.206+0000 7f4f11ffb700 1 -- 172.21.15.113:0/687885320 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4f18021070 con 0x7f4f1c073f90 2024-01-16T13:52:19.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.207+0000 7f4f11ffb700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4f0006b350 0x7f4f0006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.207+0000 7f4f11ffb700 1 -- 172.21.15.113:0/687885320 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f1808b250 con 0x7f4f1c073f90 2024-01-16T13:52:19.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.207+0000 7f4f20f47700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4f0006b350 0x7f4f0006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.207+0000 7f4f20f47700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4f0006b350 0x7f4f0006d810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c070220 tx=0x7f4f0c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.209+0000 7f4f2fba8b80 1 Processor -- start 2024-01-16T13:52:19.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.209+0000 7f4f2fba8b80 1 -- start start 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d841cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d81e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d7f8dd0 0x559d6d7fb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d76e480 con 0x559d6d786ce0 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d754770 con 0x559d6d7f8dd0 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d6d752090 con 0x559d6d823410 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d841cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d841cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48466/0 (socket says 172.21.15.113:48466) 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 -- 172.21.15.113:0/4059430774 learned_addr learned my addr 172.21.15.113:0/4059430774 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21f49700 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d7f8dd0 0x559d6d7fb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f20f47700 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d81e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 -- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 msgr2=0x559d6d81e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d81e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 -- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d7f8dd0 msgr2=0x559d6d7fb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d7f8dd0 0x559d6d7fb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 -- 172.21.15.113:0/4059430774 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d6d576b10 con 0x559d6d786ce0 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4f21748700 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d841cb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f18089830 tx=0x7f4f180597f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a73a3fc92585bbe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4efeffd700 1 -- 172.21.15.113:0/4059430774 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1808b0d0 con 0x559d6d786ce0 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4efeffd700 1 -- 172.21.15.113:0/4059430774 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1805c070 con 0x559d6d786ce0 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.210+0000 7f4efeffd700 1 -- 172.21.15.113:0/4059430774 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18060070 con 0x559d6d786ce0 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/4059430774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 msgr2=0x559d6d841cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d841cb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f18089830 tx=0x7f4f180597f0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/4059430774 shutdown_connections 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d7f8dd0 0x559d6d7fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d841cb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/4059430774 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d81e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/4059430774 >> 172.21.15.113:0/4059430774 conn(0x559d6d728410 msgr2=0x559d6d840830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/4059430774 shutdown_connections 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/4059430774 wait complete. 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.211+0000 7f4f2fba8b80 1 -- start start 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.212+0000 7f4f2fba8b80 1 Processor -- start 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.212+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d81b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.212+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d829270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.212+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d823410 0x559d6d7f9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d74d980 con 0x559d6d786ce0 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d6ed000 con 0x559d6d823410 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d6d6ecca0 con 0x559d6d7f8dd0 2024-01-16T13:52:19.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 10 client.0 _ll_register_callbacks cb 0x559d6d550350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d81b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d81b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48470/0 (socket says 172.21.15.113:48470) 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 -- 172.21.15.113:0/1251173971 learned_addr learned my addr 172.21.15.113:0/1251173971 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f20f47700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d829270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21f49700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d823410 0x559d6d7f9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 -- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 msgr2=0x559d6d829270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d829270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 -- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d823410 msgr2=0x559d6d7f9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d823410 0x559d6d7f9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 -- 172.21.15.113:0/1251173971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c0adac0 con 0x559d6d786ce0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f21748700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d81b420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c010 tx=0x7f4f1801d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bb6a39318b37b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4ef7fff700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4ef7fff700 1 -- 172.21.15.113:0/1251173971 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18060070 con 0x559d6d786ce0 2024-01-16T13:52:19.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4ef7fff700 1 -- 172.21.15.113:0/1251173971 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f18059e10 con 0x559d6d786ce0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d6d76d450 con 0x559d6d786ce0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4ef7fff700 1 -- 172.21.15.113:0/1251173971 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f1805df60 con 0x559d6d786ce0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 10 client.4629 fetch_fsmap learned FSMap version 8 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x559d6d70f290 con 0x559d6d786ce0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.213+0000 7f4ef7fff700 1 -- 172.21.15.113:0/1251173971 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559d6d76d450 con 0x559d6d786ce0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4ef7fff700 1 -- 172.21.15.113:0/1251173971 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f18089320 con 0x559d6d786ce0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f2fba8b80 10 client.4629 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f2fba8b80 10 client.4629 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f2fba8b80 20 client.4629 populate_metadata read hostname 'smithi113' 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d838c80 0x559d6d839060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f2fba8b80 4 client.4629 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 --> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d6d746960 con 0x559d6d838c80 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f20f47700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d838c80 0x559d6d839060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4f20f47700 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d838c80 0x559d6d839060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f8a20802cc44be server_cookie=522a95ba0aeed0a5 in_seq=0 out_seq=0 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.214+0000 7f4ef7fff700 10 client.4629 ms_handle_connect on v2:172.21.15.124:6832/4249021810 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4ef7fff700 1 -- 172.21.15.113:0/1251173971 <== mds.0 v2:172.21.15.124:6832/4249021810 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d6d746960 con 0x559d6d838c80 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4ef7fff700 10 client.4629 handle_command_reply: tid=0 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 client.4629 shutdown 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 msgr2=0x559d6d81b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d81b420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c010 tx=0x7f4f1801d040 comp rx=0 tx=0).stop 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 shutdown_connections 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d838c80 0x559d6d839060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d823410 0x559d6d7f9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4ef7fff700 0 client.4629 ms_handle_reset on v2:172.21.15.124:6832/4249021810 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d786ce0 0x559d6d81b420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/1251173971 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d829270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 >> 172.21.15.113:0/1251173971 conn(0x559d6d728410 msgr2=0x559d6d7f9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 shutdown_connections 2024-01-16T13:52:19.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/1251173971 wait complete. 2024-01-16T13:52:19.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.215+0000 7f4f2fba8b80 20 client.4629 trim_cache size 0 max 16384 2024-01-16T13:52:19.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.233+0000 7f4f2fba8b80 1 Processor -- start 2024-01-16T13:52:19.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.233+0000 7f4f2fba8b80 1 -- start start 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d7f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d81d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 0x559d6d839f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d5cc650 con 0x559d6d823410 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d8399a0 con 0x559d6d786ce0 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d6d8292d0 con 0x559d6d7f8dd0 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 0x559d6d839f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 0x559d6d839f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48472/0 (socket says 172.21.15.113:48472) 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 -- 172.21.15.113:0/2031223677 learned_addr learned my addr 172.21.15.113:0/2031223677 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f20f47700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d81d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21748700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d7f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 -- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 msgr2=0x559d6d81d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d81d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 -- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 msgr2=0x559d6d7f91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d7f91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 -- 172.21.15.113:0/2031223677 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d6d5cec80 con 0x559d6d823410 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f20f47700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d81d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21748700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d7f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4f21f49700 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 0x559d6d839f80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009f50 tx=0x7f4f14007610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27951c18be7b547b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4eff7fe700 1 -- 172.21.15.113:0/2031223677 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14017440 con 0x559d6d823410 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4eff7fe700 1 -- 172.21.15.113:0/2031223677 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1400f070 con 0x559d6d823410 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.234+0000 7f4eff7fe700 1 -- 172.21.15.113:0/2031223677 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f140175e0 con 0x559d6d823410 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/2031223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 msgr2=0x559d6d839f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 0x559d6d839f80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009f50 tx=0x7f4f14007610 comp rx=0 tx=0).stop 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/2031223677 shutdown_connections 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d823410 0x559d6d839f80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d7f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/2031223677 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d7f8dd0 0x559d6d81d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/2031223677 >> 172.21.15.113:0/2031223677 conn(0x559d6d728410 msgr2=0x559d6d839160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/2031223677 shutdown_connections 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/2031223677 wait complete. 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 -- start start 2024-01-16T13:52:19.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.235+0000 7f4f2fba8b80 1 Processor -- start 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d861500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 0x559d6d8a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d843cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d836f60 con 0x559d6d7f8dd0 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d6d8aa4e0 con 0x559d6d786ce0 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d6d844970 con 0x559d6d823410 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f2fba8b80 10 client.0 _ll_register_callbacks cb 0x7f4f18059a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f20f47700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 0x559d6d8a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f20f47700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 0x559d6d8a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48482/0 (socket says 172.21.15.113:48482) 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.236+0000 7f4f20f47700 1 -- 172.21.15.113:0/3227103158 learned_addr learned my addr 172.21.15.113:0/3227103158 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f21f49700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d843cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f21748700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d861500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f20f47700 1 -- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 msgr2=0x559d6d843cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f20f47700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d843cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f20f47700 1 -- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 msgr2=0x559d6d861500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f20f47700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d861500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f20f47700 1 -- 172.21.15.113:0/3227103158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f14009ba0 con 0x559d6d7f8dd0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f20f47700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 0x559d6d8a9fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c009430 tx=0x7f4f0c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecc6514fed05ee8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4ef7fff700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4ef7fff700 1 -- 172.21.15.113:0/3227103158 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c015be0 con 0x559d6d7f8dd0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4ef7fff700 1 -- 172.21.15.113:0/3227103158 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f0c010e70 con 0x559d6d7f8dd0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f1805c070 con 0x559d6d7f8dd0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4ef7fff700 1 -- 172.21.15.113:0/3227103158 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f0c015d40 con 0x559d6d7f8dd0 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 10 client.4635 fetch_fsmap learned FSMap version 8 2024-01-16T13:52:19.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x559d6d574fe0 con 0x559d6d7f8dd0 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4ef7fff700 1 -- 172.21.15.113:0/3227103158 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1805c070 con 0x559d6d7f8dd0 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4ef7fff700 1 -- 172.21.15.113:0/3227103158 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f0c032450 con 0x559d6d7f8dd0 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 10 client.4635 fetch_fsmap finished waiting for FSMap version 8 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 10 client.4635 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 20 client.4635 populate_metadata read hostname 'smithi113' 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d827550 0x559d6d827930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 4 client.4635 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.237+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 --> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d6d746680 con 0x559d6d827550 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f21748700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d827550 0x559d6d827930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4ef7fff700 10 client.4635 ms_handle_connect on v2:172.21.15.124:6832/4249021810 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f21748700 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d827550 0x559d6d827930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41b4b52376ee8f1c server_cookie=6f6bf71a74a9b7b2 in_seq=0 out_seq=0 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4ef7fff700 1 -- 172.21.15.113:0/3227103158 <== mds.0 v2:172.21.15.124:6832/4249021810 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x559d6d746680 con 0x559d6d827550 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4ef7fff700 10 client.4635 handle_command_reply: tid=0 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 client.4635 shutdown 2024-01-16T13:52:19.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 msgr2=0x559d6d8a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 0x559d6d8a9fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c009430 tx=0x7f4f0c0047c0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 shutdown_connections 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:6832/4249021810,v1:172.21.15.124:6833/4249021810] conn(0x559d6d827550 0x559d6d827930 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x559d6d823410 0x559d6d843cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4ef7fff700 0 client.4635 ms_handle_reset on v2:172.21.15.124:6832/4249021810 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x559d6d786ce0 0x559d6d861500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 --2- 172.21.15.113:0/3227103158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559d6d7f8dd0 0x559d6d8a9fa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.238+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 >> 172.21.15.113:0/3227103158 conn(0x559d6d728410 msgr2=0x559d6d7f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 shutdown_connections 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f2fba8b80 1 -- 172.21.15.113:0/3227103158 wait complete. 2024-01-16T13:52:19.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f2fba8b80 20 client.4635 trim_cache size 0 max 16384 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "cluster_fsid": "e77ab909-7680-4e53-a08d-a26ae42f1a8c", 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "whoami": 0, 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 4279, 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "want_state": "up:active", 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "up:active", 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "fs_name": "cephfs", 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "rank_uptime": 14.327052754, 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "mdsmap_epoch": 8, 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch": 21, 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch_barrier": 21, 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout: "uptime": 21.373018532 2024-01-16T13:52:19.228 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:52:19.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4f0006b350 msgr2=0x7f4f0006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4f0006b350 0x7f4f0006d810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c070220 tx=0x7f4f0c008040 comp rx=0 tx=0).stop 2024-01-16T13:52:19.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 msgr2=0x7f4f1c158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 0x7f4f1c158880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4f18000c00 tx=0x7f4f18004750 comp rx=0 tx=0).stop 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 shutdown_connections 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4f0006b350 0x7f4f0006d810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f1c15e2a0 0x7f4f1c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c073f90 0x7f4f1c158880 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 --2- 172.21.15.113:0/687885320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f1c0ce090 0x7f4f1c15ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 >> 172.21.15.113:0/687885320 conn(0x7f4f1c0cd570 msgr2=0x7f4f1c06b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 shutdown_connections 2024-01-16T13:52:19.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.239+0000 7f4f239ac700 1 -- 172.21.15.113:0/687885320 wait complete. 2024-01-16T13:52:19.233 INFO:teuthology.run_tasks:Running task kclient... 2024-01-16T13:52:19.363 INFO:tasks.kclient:Mounting kernel clients... 2024-01-16T13:52:19.363 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-16T13:52:19.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.363 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-16T13:52:19.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.381 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-01-16T13:52:19.449 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-01-16T13:52:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:52:19.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.451 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-16T13:52:19.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-01-16T13:52:19.576 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-01-16T13:52:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:52:19.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.578 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2024-01-16T13:52:19.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.599 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2024-01-16T13:52:19.675 INFO:teuthology.orchestra.run.smithi124.stderr:Cannot find device "ceph-brx" 2024-01-16T13:52:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:52:19.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.677 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2024-01-16T13:52:19.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:19.738 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2024-01-16T13:52:19.812 INFO:teuthology.orchestra.run.smithi124.stderr:Cannot find device "ceph-brx" 2024-01-16T13:52:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T13:52:19.814 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-01-16T13:52:19.814 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T13:52:19.814 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T13:52:19.814 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T13:52:19.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.956+0000 7fc14292a700 1 Processor -- start 2024-01-16T13:52:19.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.956+0000 7fc14292a700 1 -- start start 2024-01-16T13:52:19.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc14292a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c15e2a0 0x7fc13c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc14292a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc13c0ce090 0x7fc13c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc14292a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c06e7f0 con 0x7fc13c073f90 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc14292a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c06dec0 con 0x7fc13c0ce090 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc14292a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c06e1a0 con 0x7fc13c15e2a0 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc13c0ce090 0x7fc13c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc13c0ce090 0x7fc13c06ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48484/0 (socket says 172.21.15.113:48484) 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 -- 172.21.15.113:0/1693810433 learned_addr learned my addr 172.21.15.113:0/1693810433 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 -- 172.21.15.113:0/1693810433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c15e2a0 msgr2=0x7fc13c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 --2- 172.21.15.113:0/1693810433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c15e2a0 0x7fc13c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 -- 172.21.15.113:0/1693810433 >> v1:172.21.15.124:6789/0 conn(0x7fc13c073f90 legacy=0x7fc13c0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 -- 172.21.15.113:0/1693810433 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c06e480 con 0x7fc13c0ce090 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.957+0000 7fc13b7fe700 1 --2- 172.21.15.113:0/1693810433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c0ce090 0x7fc13c06ec80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0098f0 tx=0x7fc12c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942475c5fad0c0bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc13affd700 1 -- 172.21.15.113:0/1693810433 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c004230 con 0x7fc13c0ce090 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc13affd700 1 -- 172.21.15.113:0/1693810433 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c004390 con 0x7fc13c0ce090 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc13affd700 1 -- 172.21.15.113:0/1693810433 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c0036a0 con 0x7fc13c0ce090 2024-01-16T13:52:19.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 -- 172.21.15.113:0/1693810433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c0ce090 msgr2=0x7fc13c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 --2- 172.21.15.113:0/1693810433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c0ce090 0x7fc13c06ec80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0098f0 tx=0x7fc12c009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 -- 172.21.15.113:0/1693810433 shutdown_connections 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 --2- 172.21.15.113:0/1693810433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c15e2a0 0x7fc13c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 --2- 172.21.15.113:0/1693810433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c0ce090 0x7fc13c06ec80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 -- 172.21.15.113:0/1693810433 >> 172.21.15.113:0/1693810433 conn(0x7fc13c0cd570 msgr2=0x7fc13c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 -- 172.21.15.113:0/1693810433 shutdown_connections 2024-01-16T13:52:19.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.958+0000 7fc14292a700 1 -- 172.21.15.113:0/1693810433 wait complete. 2024-01-16T13:52:19.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.959+0000 7fc14292a700 1 Processor -- start 2024-01-16T13:52:19.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 -- start start 2024-01-16T13:52:19.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 0x7fc13c159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c0ce090 0x7fc13c161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc13c15e2a0 0x7fc13c15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c159e50 con 0x7fc13c073f90 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1617b0 con 0x7fc13c15e2a0 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc14292a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c1601d0 con 0x7fc13c0ce090 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc13bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 0x7fc13c159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc13bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 0x7fc13c159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48490/0 (socket says 172.21.15.113:48490) 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.960+0000 7fc13bfff700 1 -- 172.21.15.113:0/3120429245 learned_addr learned my addr 172.21.15.113:0/3120429245 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13b7fe700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c0ce090 0x7fc13c161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc140ec7700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc13c15e2a0 0x7fc13c15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13bfff700 1 -- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c0ce090 msgr2=0x7fc13c161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13bfff700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c0ce090 0x7fc13c161270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13bfff700 1 -- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc13c15e2a0 msgr2=0x7fc13c15a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13bfff700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc13c15e2a0 0x7fc13c15a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:19.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13bfff700 1 -- 172.21.15.113:0/3120429245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c009550 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc13bfff700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 0x7fc13c159910 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b530 tx=0x7fc13000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130011830 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1300119b0 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c1604b0 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130011d60 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.961+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c160a70 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.962+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c067f60 con 0x7fc13c073f90 2024-01-16T13:52:19.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.962+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc13001a400 con 0x7fc13c073f90 2024-01-16T13:52:19.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.963+0000 7fc138ff9700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc12806b350 0x7fc12806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:19.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.963+0000 7fc13b7fe700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc12806b350 0x7fc12806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:19.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.963+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc130089c20 con 0x7fc13c073f90 2024-01-16T13:52:19.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.963+0000 7fc13b7fe700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc12806b350 0x7fc12806d810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc13c15fdc0 tx=0x7fc12c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:19.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:19.967+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc130054b20 con 0x7fc13c073f90 2024-01-16T13:52:20.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.200+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc13c072e00 con 0x7fc13c073f90 2024-01-16T13:52:20.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.200+0000 7fc138ff9700 1 -- 172.21.15.113:0/3120429245 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc130015020 con 0x7fc13c073f90 2024-01-16T13:52:20.182 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.202+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc12806b350 msgr2=0x7fc12806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.202+0000 7fc14292a700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc12806b350 0x7fc12806d810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc13c15fdc0 tx=0x7fc12c008030 comp rx=0 tx=0).stop 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.202+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 msgr2=0x7fc13c159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.202+0000 7fc14292a700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 0x7fc13c159910 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b530 tx=0x7fc13000b8f0 comp rx=0 tx=0).stop 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 shutdown_connections 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc12806b350 0x7fc12806d810 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc13c15e2a0 0x7fc13c15a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:20.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc13c073f90 0x7fc13c159910 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:20.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 --2- 172.21.15.113:0/3120429245 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc13c0ce090 0x7fc13c161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:20.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 >> 172.21.15.113:0/3120429245 conn(0x7fc13c0cd570 msgr2=0x7fc13c0b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:20.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 shutdown_connections 2024-01-16T13:52:20.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:20.203+0000 7fc14292a700 1 -- 172.21.15.113:0/3120429245 wait complete. 2024-01-16T13:52:20.196 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T13:52:20.197 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T13:52:20.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.197 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-01-16T13:52:20.214 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:20.215 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T13:52:20.216 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:20.216 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T13:52:20.216 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42485sec preferred_lft 42485sec 2024-01-16T13:52:20.216 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::9d38:c2c1:4a69:a61c/64 scope link noprefixroute 2024-01-16T13:52:20.216 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:20.216 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T13:52:20.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.216 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:20.216 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:20.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add name ceph-brx type bridge 2024-01-16T13:52:20.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr flush dev ceph-brx 2024-01-16T13:52:20.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set ceph-brx up 2024-01-16T13:52:20.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T13:52:20.217 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:20.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.390 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T13:52:20.418 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-01-16T13:52:20.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.420 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-01-16T13:52:20.493 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-01-16T13:52:20.493 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T13:52:20.493 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T13:52:20.494 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T13:52:20.494 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T13:52:20.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.494 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:20.494 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:20.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T13:52:20.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T13:52:20.494 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T13:52:20.494 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:20.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.705 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-16T13:52:20.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.722 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-01-16T13:52:20.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.779 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:20.779 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:20.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T13:52:20.779 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-16T13:52:20.779 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:20.875 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-16T13:52:20.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-16T13:52:20.875 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:21.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.077 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:21.077 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:21.077 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.0 up 2024-01-16T13:52:21.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T13:52:21.078 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:21.175 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:21.175 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T13:52:21.192 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T13:52:21.192 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:21.192 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T13:52:21.248 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-16T13:52:21.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.248 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:52:21.335 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:52:21.335 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-16T13:52:21.335 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:52:21.443 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:52:21.443 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-16T13:52:21.444 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:52:21.444 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:21.444 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:21.462 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:52:21.462 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-16T13:52:21.462 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:52:21.462 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:21.462 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:21.462 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790" 2024-01-16T13:52:21.463 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2024-01-16T13:52:21.463 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e77ab909-7680-4e53-a08d-a26ae42f1a8c" will pass to kernel 2024-01-16T13:52:21.465 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-16T13:52:21.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.465 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T13:52:21.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.493 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:21.582 INFO:teuthology.orchestra.run.smithi113.stdout:client4644 2024-01-16T13:52:21.582 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:21.582 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:21.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.599 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:21.687 INFO:teuthology.orchestra.run.smithi113.stdout:client4644 2024-01-16T13:52:21.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.688 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:21.735 INFO:teuthology.orchestra.run.smithi113.stdout:client4644 2024-01-16T13:52:21.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:21.735 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e77ab909-7680-4e53-a08d-a26ae42f1a8c.client4644/status') 2024-01-16T13:52:21.788 INFO:teuthology.orchestra.run.smithi113.stdout:instance: client.4644 (3)192.168.0.1:0/83437776 2024-01-16T13:52:21.788 INFO:teuthology.orchestra.run.smithi113.stdout:blocklisted: false 2024-01-16T13:52:21.788 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records in 2024-01-16T13:52:21.788 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records out 2024-01-16T13:52:21.788 INFO:teuthology.orchestra.run.smithi113.stderr:67 bytes copied, 2.0132e-05 s, 3.3 MB/s 2024-01-16T13:52:21.790 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-01-16T13:52:21.790 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T13:52:21.790 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T13:52:21.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T13:52:21.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.929+0000 7f5dbdfb9700 1 Processor -- start 2024-01-16T13:52:21.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.929+0000 7f5dbdfb9700 1 -- start start 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5dbdfb9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5db80b59f0 0x7f5db806db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5dbdfb9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db80b4350 0x7f5db80af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5dbdfb9700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80b0290 con 0x7f5db806e0b0 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5dbdfb9700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db80b4840 con 0x7f5db80b59f0 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5dbdfb9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80b4b00 con 0x7f5db80b4350 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5db80b59f0 0x7f5db806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5db80b59f0 0x7f5db806db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48496/0 (socket says 172.21.15.113:48496) 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 -- 172.21.15.113:0/1586398457 learned_addr learned my addr 172.21.15.113:0/1586398457 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 -- 172.21.15.113:0/1586398457 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db80b4350 msgr2=0x7f5db80af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:21.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 --2- 172.21.15.113:0/1586398457 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db80b4350 0x7f5db80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 -- 172.21.15.113:0/1586398457 >> v1:172.21.15.124:6789/0 conn(0x7f5db806e0b0 legacy=0x7f5db80b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 -- 172.21.15.113:0/1586398457 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80ae9b0 con 0x7f5db80b59f0 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.930+0000 7f5db77fe700 1 --2- 172.21.15.113:0/1586398457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b59f0 0x7f5db806db70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009920 tx=0x7f5dac009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ab1990498cc6a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5db67fc700 1 -- 172.21.15.113:0/1586398457 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac004230 con 0x7f5db80b59f0 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5db67fc700 1 -- 172.21.15.113:0/1586398457 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac004390 con 0x7f5db80b59f0 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5db67fc700 1 -- 172.21.15.113:0/1586398457 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac009e70 con 0x7f5db80b59f0 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1586398457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b59f0 msgr2=0x7f5db806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1586398457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b59f0 0x7f5db806db70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009920 tx=0x7f5dac009c30 comp rx=0 tx=0).stop 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1586398457 shutdown_connections 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1586398457 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db80b4350 0x7f5db80af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:21.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1586398457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b59f0 0x7f5db806db70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:21.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1586398457 >> 172.21.15.113:0/1586398457 conn(0x7f5db8072990 msgr2=0x7f5db806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:21.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1586398457 shutdown_connections 2024-01-16T13:52:21.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.931+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1586398457 wait complete. 2024-01-16T13:52:21.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.933+0000 7f5dbdfb9700 1 Processor -- start 2024-01-16T13:52:21.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.933+0000 7f5dbdfb9700 1 -- start start 2024-01-16T13:52:21.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db806e0b0 0x7f5db80cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:21.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 0x7f5db813ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:21.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db80b59f0 0x7f5db80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db813e310 con 0x7f5db80b4350 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db813e5b0 con 0x7f5db80b59f0 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80cf3a0 con 0x7f5db806e0b0 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 0x7f5db813ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 0x7f5db813ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48504/0 (socket says 172.21.15.113:48504) 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 -- 172.21.15.113:0/1449842114 learned_addr learned my addr 172.21.15.113:0/1449842114 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db77fe700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db806e0b0 0x7f5db80cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db7fff700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db80b59f0 0x7f5db80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 -- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db806e0b0 msgr2=0x7f5db80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db806e0b0 0x7f5db80cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 -- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db80b59f0 msgr2=0x7f5db80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db80b59f0 0x7f5db80cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 -- 172.21.15.113:0/1449842114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac009550 con 0x7f5db80b4350 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db77fe700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db806e0b0 0x7f5db80cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:21.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db7fff700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db80b59f0 0x7f5db80cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5db6ffd700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 0x7f5db813ddd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5da800c4a0 tx=0x7f5da800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da80054e0 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da8003680 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db80aed40 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da8003a10 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.934+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db80cf960 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.935+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db8067f60 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.936+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5da8003b70 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.936+0000 7f5d9ffff700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5da406b380 0x7f5da406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.936+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5da8088ff0 con 0x7f5db80b4350 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.936+0000 7f5db77fe700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5da406b380 0x7f5da406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:21.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.937+0000 7f5db77fe700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5da406b380 0x7f5da406d840 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009130 tx=0x7f5dac008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:21.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:21.941+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5da8053e80 con 0x7f5db80b4350 2024-01-16T13:52:22.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.175+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5db806ec40 con 0x7f5db80b4350 2024-01-16T13:52:22.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.176+0000 7f5d9ffff700 1 -- 172.21.15.113:0/1449842114 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5da8017020 con 0x7f5db80b4350 2024-01-16T13:52:22.158 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5da406b380 msgr2=0x7f5da406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5da406b380 0x7f5da406d840 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009130 tx=0x7f5dac008030 comp rx=0 tx=0).stop 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 msgr2=0x7f5db813ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 0x7f5db813ddd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5da800c4a0 tx=0x7f5da800c860 comp rx=0 tx=0).stop 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 shutdown_connections 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5da406b380 0x7f5da406d840 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db80b59f0 0x7f5db80cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:22.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5db806e0b0 0x7f5db80cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:22.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 --2- 172.21.15.113:0/1449842114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5db80b4350 0x7f5db813ddd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:22.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 >> 172.21.15.113:0/1449842114 conn(0x7f5db8072990 msgr2=0x7f5db8161c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:22.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 shutdown_connections 2024-01-16T13:52:22.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:22.178+0000 7f5dbdfb9700 1 -- 172.21.15.113:0/1449842114 wait complete. 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T13:52:22.172 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-16T13:52:22.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.173 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-01-16T13:52:22.190 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:22.190 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:22.190 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:22.190 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.190 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T13:52:22.191 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42483sec preferred_lft 42483sec 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::9d38:c2c1:4a69:a61c/64 scope link noprefixroute 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 62:af:26:5d:98:8b brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::866:67ff:fe19:1e59/64 scope link tentative 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 62:af:26:5d:98:8b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::60af:26ff:fe5d:988b/64 scope link tentative 2024-01-16T13:52:22.192 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.193 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T13:52:22.256 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-01-16T13:52:22.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.258 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-01-16T13:52:22.316 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-01-16T13:52:22.316 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T13:52:22.317 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T13:52:22.317 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T13:52:22.317 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T13:52:22.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.317 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:22.317 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:22.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T13:52:22.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T13:52:22.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T13:52:22.318 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:22.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.422 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-16T13:52:22.439 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T13:52:22.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.440 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-01-16T13:52:22.496 INFO:teuthology.orchestra.run.smithi113.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-16T13:52:22.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.497 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:22.497 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:22.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T13:52:22.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-16T13:52:22.497 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:22.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.593 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T13:52:22.634 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 9a:7a:cb:b9:b4:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::987a:cbff:feb9:b4a2/64 scope link 2024-01-16T13:52:22.635 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.647 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 9a:7a:cb:b9:b4:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T13:52:22.722 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T13:52:22.723 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.723 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::987a:cbff:feb9:b4a2/64 scope link 2024-01-16T13:52:22.723 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:22.738 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-16T13:52:22.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.738 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:22.738 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:22.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-16T13:52:22.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T13:52:22.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-16T13:52:22.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-16T13:52:22.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-16T13:52:22.739 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:22.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:22.996 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:52:22.996 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:52:22.996 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.1 up 2024-01-16T13:52:22.996 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T13:52:22.996 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:52:23.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:23.093 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-16T13:52:23.110 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-16T13:52:23.110 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:23.110 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-16T13:52:23.166 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-16T13:52:23.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:23.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-16T13:52:23.252 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-16T13:52:23.252 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-16T13:52:23.252 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:52:23.261 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-16T13:52:23.261 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-16T13:52:23.261 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:52:23.261 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:23.261 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790" 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2024-01-16T13:52:23.280 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=e77ab909-7680-4e53-a08d-a26ae42f1a8c" will pass to kernel 2024-01-16T13:52:23.282 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-16T13:52:23.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:23.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-16T13:52:23.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:23.309 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:23.382 INFO:teuthology.orchestra.run.smithi113.stdout:client4606 2024-01-16T13:52:23.382 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:23.383 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:23.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:23.440 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:23.513 INFO:teuthology.orchestra.run.smithi113.stdout:client4606 2024-01-16T13:52:23.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:23.513 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:23.585 INFO:teuthology.orchestra.run.smithi113.stdout:client4606 2024-01-16T13:52:23.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:23.586 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e77ab909-7680-4e53-a08d-a26ae42f1a8c.client4606/status') 2024-01-16T13:52:23.666 INFO:teuthology.orchestra.run.smithi113.stdout:instance: client.4606 (3)172.21.15.113:0/3401599286 2024-01-16T13:52:23.666 INFO:teuthology.orchestra.run.smithi113.stdout:blocklisted: false 2024-01-16T13:52:23.666 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records in 2024-01-16T13:52:23.666 INFO:teuthology.orchestra.run.smithi113.stderr:0+1 records out 2024-01-16T13:52:23.666 INFO:teuthology.orchestra.run.smithi113.stderr:71 bytes copied, 3.8529e-05 s, 1.8 MB/s 2024-01-16T13:52:23.667 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-01-16T13:52:23.668 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T13:52:23.668 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T13:52:23.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T13:52:23.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.847+0000 7f7576cc1700 1 Processor -- start 2024-01-16T13:52:23.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.847+0000 7f7576cc1700 1 -- start start 2024-01-16T13:52:23.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.847+0000 7f7576cc1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f75701473a0 0x7f757015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:23.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f7576cc1700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f757006dc20 0x7f7570073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:23.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f7576cc1700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757015eb00 con 0x7f7570147780 2024-01-16T13:52:23.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f7576cc1700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757015ede0 con 0x7f757006dc20 2024-01-16T13:52:23.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f7576cc1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757015f0c0 con 0x7f75701473a0 2024-01-16T13:52:23.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f757525e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f757006dc20 0x7f7570073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f756ffff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7570147780 0x7f757006c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55968/0 (socket says 172.21.15.113:55968) 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f756ffff700 1 -- 172.21.15.113:0/417702481 learned_addr learned my addr 172.21.15.113:0/417702481 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f7574a5d700 1 --2- 172.21.15.113:0/417702481 >> v2:172.21.15.124:3300/0 conn(0x7f75701473a0 0x7f757015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f757525e700 1 -- 172.21.15.113:0/417702481 >> v2:172.21.15.124:3300/0 conn(0x7f75701473a0 msgr2=0x7f757015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f757525e700 1 --2- 172.21.15.113:0/417702481 >> v2:172.21.15.124:3300/0 conn(0x7f75701473a0 0x7f757015e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f757525e700 1 -- 172.21.15.113:0/417702481 >> v1:172.21.15.113:6789/0 conn(0x7f7570147780 legacy=0x7f757006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f757525e700 1 -- 172.21.15.113:0/417702481 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757015f3a0 con 0x7f757006dc20 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f756f7fe700 1 -- 172.21.15.113:0/417702481 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163115379 0 0) 0x7f757015eb00 con 0x7f7570147780 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f757525e700 1 --2- 172.21.15.113:0/417702481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f757006dc20 0x7f7570073f90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7570072f90 tx=0x7f756800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48bb8ab6fe68e95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f756f7fe700 1 -- 172.21.15.113:0/417702481 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756800e070 con 0x7f757006dc20 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f756f7fe700 1 -- 172.21.15.113:0/417702481 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756800bbd0 con 0x7f757006dc20 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.848+0000 7f756f7fe700 1 -- 172.21.15.113:0/417702481 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568012430 con 0x7f757006dc20 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 -- 172.21.15.113:0/417702481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f757006dc20 msgr2=0x7f7570073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:23.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 --2- 172.21.15.113:0/417702481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f757006dc20 0x7f7570073f90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7570072f90 tx=0x7f756800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 -- 172.21.15.113:0/417702481 shutdown_connections 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 --2- 172.21.15.113:0/417702481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f757006dc20 0x7f7570073f90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 --2- 172.21.15.113:0/417702481 >> v2:172.21.15.124:3300/0 conn(0x7f75701473a0 0x7f757015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 -- 172.21.15.113:0/417702481 >> 172.21.15.113:0/417702481 conn(0x7f75700b3130 msgr2=0x7f75700d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 -- 172.21.15.113:0/417702481 shutdown_connections 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.849+0000 7f7576cc1700 1 -- 172.21.15.113:0/417702481 wait complete. 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.850+0000 7f7576cc1700 1 Processor -- start 2024-01-16T13:52:23.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.850+0000 7f7576cc1700 1 -- start start 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7576cc1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f757006dc20 0x7f75701f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7576cc1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f75701473a0 0x7f75700cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7576cc1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 0x7f75700cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7576cc1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75701f8350 con 0x7f7570147780 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7576cc1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75701f73c0 con 0x7f757006dc20 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7576cc1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700ce030 con 0x7f75701473a0 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 0x7f75700cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 0x7f75700cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41534/0 (socket says 172.21.15.113:41534) 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 -- 172.21.15.113:0/4208301596 learned_addr learned my addr 172.21.15.113:0/4208301596 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f7574a5d700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f757006dc20 0x7f75701f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:23.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f756ffff700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f75701473a0 0x7f75700cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 -- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f75701473a0 msgr2=0x7f75700cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f75701473a0 0x7f75700cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 -- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f757006dc20 msgr2=0x7f75701f7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f757006dc20 0x7f75701f7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.851+0000 7f757525e700 1 -- 172.21.15.113:0/4208301596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568009a70 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.852+0000 7f757525e700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 0x7f75700cd570 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7570074d90 tx=0x7f7568004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.852+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568012430 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.852+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7568020070 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.852+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700ce310 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.852+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568004280 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.852+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75701f78f0 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.853+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7570146c00 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.853+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f75680043e0 con 0x7f7570147780 2024-01-16T13:52:23.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.853+0000 7f756d7fa700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f755406b350 0x7f755406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:23.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.854+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7568022070 con 0x7f7570147780 2024-01-16T13:52:23.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.854+0000 7f7574a5d700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f755406b350 0x7f755406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:23.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.854+0000 7f7574a5d700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f755406b350 0x7f755406d810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f75640099d0 tx=0x7f7564008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:23.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:23.858+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7568056c00 con 0x7f7570147780 2024-01-16T13:52:24.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.090+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7570067f60 con 0x7f7570147780 2024-01-16T13:52:24.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.090+0000 7f756d7fa700 1 -- 172.21.15.113:0/4208301596 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f756805a220 con 0x7f7570147780 2024-01-16T13:52:24.073 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.092+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f755406b350 msgr2=0x7f755406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.092+0000 7f7576cc1700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f755406b350 0x7f755406d810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f75640099d0 tx=0x7f7564008040 comp rx=0 tx=0).stop 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 msgr2=0x7f75700cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 0x7f75700cd570 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7570074d90 tx=0x7f7568004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 shutdown_connections 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f755406b350 0x7f755406d810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7570147780 0x7f75700cd570 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f757006dc20 0x7f75701f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 --2- 172.21.15.113:0/4208301596 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f75701473a0 0x7f75700cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:24.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 >> 172.21.15.113:0/4208301596 conn(0x7f75700b3130 msgr2=0x7f75701618a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 shutdown_connections 2024-01-16T13:52:24.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:24.093+0000 7f7576cc1700 1 -- 172.21.15.113:0/4208301596 wait complete. 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T13:52:24.087 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T13:52:24.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.088 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2024-01-16T13:52:24.109 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42451sec preferred_lft 42451sec 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec99:5e7d:854c:8532/64 scope link noprefixroute 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:24.110 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T13:52:24.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add name ceph-brx type bridge 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip addr flush dev ceph-brx 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set ceph-brx up 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T13:52:24.111 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:24.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.291 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T13:52:24.319 INFO:teuthology.orchestra.run.smithi124.stdout:1 2024-01-16T13:52:24.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.321 DEBUG:teuthology.orchestra.run.smithi124:> route 2024-01-16T13:52:24.401 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2024-01-16T13:52:24.401 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T13:52:24.401 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T13:52:24.401 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T13:52:24.401 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T13:52:24.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.401 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:24.402 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:24.402 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T13:52:24.402 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T13:52:24.402 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T13:52:24.402 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:24.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.632 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2024-01-16T13:52:24.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.650 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list-id 2024-01-16T13:52:24.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.707 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:24.707 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:24.707 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T13:52:24.707 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-16T13:52:24.707 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:24.801 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-16T13:52:24.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:24.801 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-16T13:52:24.802 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:25.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.023 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:25.023 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:25.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2024-01-16T13:52:25.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T13:52:25.024 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:25.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:25.121 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T13:52:25.138 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T13:52:25.138 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:25.138 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T13:52:25.194 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-16T13:52:25.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.195 DEBUG:teuthology.orchestra.run.smithi124:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:52:25.281 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:52:25.281 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-16T13:52:25.281 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:52:25.386 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:52:25.387 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-16T13:52:25.387 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:52:25.387 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:25.387 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790" 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2024-01-16T13:52:25.405 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=e77ab909-7680-4e53-a08d-a26ae42f1a8c" will pass to kernel 2024-01-16T13:52:25.408 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-16T13:52:25.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.408 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T13:52:25.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.434 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:25.507 INFO:teuthology.orchestra.run.smithi124.stdout:client4615 2024-01-16T13:52:25.507 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:25.507 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:25.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.565 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:25.637 INFO:teuthology.orchestra.run.smithi124.stdout:client4615 2024-01-16T13:52:25.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.637 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:25.710 INFO:teuthology.orchestra.run.smithi124.stdout:client4615 2024-01-16T13:52:25.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:25.710 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e77ab909-7680-4e53-a08d-a26ae42f1a8c.client4615/status') 2024-01-16T13:52:25.791 INFO:teuthology.orchestra.run.smithi124.stdout:instance: client.4615 (3)192.168.0.1:0/1811858185 2024-01-16T13:52:25.791 INFO:teuthology.orchestra.run.smithi124.stdout:blocklisted: false 2024-01-16T13:52:25.792 INFO:teuthology.orchestra.run.smithi124.stderr:0+1 records in 2024-01-16T13:52:25.792 INFO:teuthology.orchestra.run.smithi124.stderr:0+1 records out 2024-01-16T13:52:25.792 INFO:teuthology.orchestra.run.smithi124.stderr:69 bytes copied, 3.4344e-05 s, 2.0 MB/s 2024-01-16T13:52:25.793 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-01-16T13:52:25.793 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T13:52:25.793 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T13:52:25.793 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T13:52:25.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.946+0000 7f0189a65700 1 Processor -- start 2024-01-16T13:52:25.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.946+0000 7f0189a65700 1 -- start start 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f0189a65700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f018415af10 0x7f018415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f0189a65700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01841414c0 con 0x7f018415f630 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f0189a65700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01840ce9f0 con 0x7f01840ce140 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f0189a65700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f018415c300 con 0x7f018415af10 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f0183fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f018415f630 0x7f018406b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55982/0 (socket says 172.21.15.113:55982) 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01837fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f018415af10 0x7f018415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f0183fff700 1 -- 172.21.15.113:0/3721879443 learned_addr learned my addr 172.21.15.113:0/3721879443 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01827fc700 1 -- 172.21.15.113:0/3721879443 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603301658 0 0) 0x7f01841414c0 con 0x7f018415f630 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01827fc700 1 -- 172.21.15.113:0/3721879443 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f016c0035b0 con 0x7f018415f630 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01837fe700 1 -- 172.21.15.113:0/3721879443 >> v1:172.21.15.124:6789/0 conn(0x7f01840ce140 legacy=0x7f018415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01837fe700 1 -- 172.21.15.113:0/3721879443 >> v1:172.21.15.113:6789/0 conn(0x7f018415f630 legacy=0x7f018406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01837fe700 1 -- 172.21.15.113:0/3721879443 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01840cecd0 con 0x7f018415af10 2024-01-16T13:52:25.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.947+0000 7f01837fe700 1 --2- 172.21.15.113:0/3721879443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f018415af10 0x7f018415e9e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0178009840 tx=0x7f0178009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc177aaf629519f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f01827fc700 1 -- 172.21.15.113:0/3721879443 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178004230 con 0x7f018415af10 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f01827fc700 1 -- 172.21.15.113:0/3721879443 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0178004390 con 0x7f018415af10 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f01827fc700 1 -- 172.21.15.113:0/3721879443 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01780036c0 con 0x7f018415af10 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 -- 172.21.15.113:0/3721879443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f018415af10 msgr2=0x7f018415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 --2- 172.21.15.113:0/3721879443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f018415af10 0x7f018415e9e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0178009840 tx=0x7f0178009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 -- 172.21.15.113:0/3721879443 shutdown_connections 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 --2- 172.21.15.113:0/3721879443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f018415af10 0x7f018415e9e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 -- 172.21.15.113:0/3721879443 >> 172.21.15.113:0/3721879443 conn(0x7f01840cd570 msgr2=0x7f01840cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:25.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 -- 172.21.15.113:0/3721879443 shutdown_connections 2024-01-16T13:52:25.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.948+0000 7f0189a65700 1 -- 172.21.15.113:0/3721879443 wait complete. 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.949+0000 7f0189a65700 1 Processor -- start 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 -- start start 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f01840ce140 0x7f0184156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f018415af10 0x7f018413ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 0x7f018415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01840cb210 con 0x7f01841404a0 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018415d570 con 0x7f018415af10 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0189a65700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018415d810 con 0x7f01840ce140 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0183fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 0x7f018415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0183fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 0x7f018415b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41546/0 (socket says 172.21.15.113:41546) 2024-01-16T13:52:25.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.950+0000 7f0183fff700 1 -- 172.21.15.113:0/2979477362 learned_addr learned my addr 172.21.15.113:0/2979477362 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0182ffd700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f018415af10 0x7f018413ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f01837fe700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f01840ce140 0x7f0184156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0183fff700 1 -- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f01840ce140 msgr2=0x7f0184156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0183fff700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f01840ce140 0x7f0184156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0183fff700 1 -- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f018415af10 msgr2=0x7f018413ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0183fff700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f018415af10 0x7f018413ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0183fff700 1 -- 172.21.15.113:0/2979477362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01780094a0 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0183fff700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 0x7f018415b2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f017c00caa0 tx=0x7f017c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c00a9e0 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c00ab40 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018415daf0 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c012400 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.951+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018415e0b0 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.952+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018415e420 con 0x7f01841404a0 2024-01-16T13:52:25.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.953+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f017c01b3b0 con 0x7f01841404a0 2024-01-16T13:52:25.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.953+0000 7f016bfff700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f017006b350 0x7f017006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:25.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.953+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f017c089750 con 0x7f01841404a0 2024-01-16T13:52:25.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.953+0000 7f01837fe700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f017006b350 0x7f017006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:25.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.953+0000 7f01837fe700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f017006b350 0x7f017006d810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01780090b0 tx=0x7f0178010000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:25.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:25.957+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f017c054710 con 0x7f01841404a0 2024-01-16T13:52:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.193+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0184071690 con 0x7f01841404a0 2024-01-16T13:52:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.194+0000 7f016bfff700 1 -- 172.21.15.113:0/2979477362 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f017c017060 con 0x7f01841404a0 2024-01-16T13:52:26.176 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T13:52:26.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f017006b350 msgr2=0x7f017006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:26.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f017006b350 0x7f017006d810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01780090b0 tx=0x7f0178010000 comp rx=0 tx=0).stop 2024-01-16T13:52:26.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 msgr2=0x7f018415b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 0x7f018415b2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f017c00caa0 tx=0x7f017c00cdb0 comp rx=0 tx=0).stop 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 shutdown_connections 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f017006b350 0x7f017006d810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01841404a0 0x7f018415b2f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f01840ce140 0x7f0184156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 --2- 172.21.15.113:0/2979477362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f018415af10 0x7f018413ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 >> 172.21.15.113:0/2979477362 conn(0x7f01840cd570 msgr2=0x7f01840cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 shutdown_connections 2024-01-16T13:52:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:26.196+0000 7f0189a65700 1 -- 172.21.15.113:0/2979477362 wait complete. 2024-01-16T13:52:26.190 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T13:52:26.190 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T13:52:26.190 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2024-01-16T13:52:26.190 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-16T13:52:26.190 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T13:52:26.191 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T13:52:26.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T13:52:26.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T13:52:26.191 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-16T13:52:26.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.191 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.211 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42449sec preferred_lft 42449sec 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec99:5e7d:854c:8532/64 scope link noprefixroute 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.212 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0e:6d:52:8c:06:03 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::6c76:17ff:fef4:cd32/64 scope link 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0e:6d:52:8c:06:03 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::c6d:52ff:fe8c:603/64 scope link tentative 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.213 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T13:52:26.280 INFO:teuthology.orchestra.run.smithi124.stdout:1 2024-01-16T13:52:26.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.281 DEBUG:teuthology.orchestra.run.smithi124:> route 2024-01-16T13:52:26.341 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2024-01-16T13:52:26.341 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T13:52:26.342 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T13:52:26.342 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T13:52:26.342 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T13:52:26.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.342 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:26.342 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:26.342 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T13:52:26.342 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T13:52:26.342 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T13:52:26.342 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:26.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.451 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2024-01-16T13:52:26.468 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T13:52:26.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.469 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list-id 2024-01-16T13:52:26.526 INFO:teuthology.orchestra.run.smithi124.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-16T13:52:26.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.526 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:26.526 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:26.526 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T13:52:26.526 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-16T13:52:26.526 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:26.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.620 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether ba:32:37:d3:a1:8c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::b832:37ff:fed3:a18c/64 scope link 2024-01-16T13:52:26.660 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.672 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T13:52:26.752 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether ba:32:37:d3:a1:8c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::b832:37ff:fed3:a18c/64 scope link 2024-01-16T13:52:26.753 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:52:26.769 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-16T13:52:26.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-16T13:52:26.769 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:26.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:26.995 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:52:26.995 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:52:26.996 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.1 up 2024-01-16T13:52:26.996 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T13:52:26.996 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:52:27.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:27.093 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-16T13:52:27.109 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-16T13:52:27.109 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:52:27.110 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-16T13:52:27.166 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-16T13:52:27.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:27.166 DEBUG:teuthology.orchestra.run.smithi124:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-16T13:52:27.253 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-16T13:52:27.253 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-16T13:52:27.253 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:52:27.262 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-16T13:52:27.262 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-16T13:52:27.262 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:52:27.262 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:27.262 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:27.280 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-01-16T13:52:27.280 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-01-16T13:52:27.280 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:52:27.280 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:52:27.280 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:52:27.280 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790" 2024-01-16T13:52:27.281 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2024-01-16T13:52:27.281 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=e77ab909-7680-4e53-a08d-a26ae42f1a8c" will pass to kernel 2024-01-16T13:52:27.283 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-16T13:52:27.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:27.283 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-16T13:52:27.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:27.309 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:27.382 INFO:teuthology.orchestra.run.smithi124.stdout:client4499 2024-01-16T13:52:27.382 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:27.383 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:27.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:27.440 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:27.512 INFO:teuthology.orchestra.run.smithi124.stdout:client4499 2024-01-16T13:52:27.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:27.512 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:52:27.585 INFO:teuthology.orchestra.run.smithi124.stdout:client4499 2024-01-16T13:52:27.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:27.585 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e77ab909-7680-4e53-a08d-a26ae42f1a8c.client4499/status') 2024-01-16T13:52:27.666 INFO:teuthology.orchestra.run.smithi124.stdout:instance: client.4499 (3)192.168.0.2:0/749169138 2024-01-16T13:52:27.666 INFO:teuthology.orchestra.run.smithi124.stdout:blocklisted: false 2024-01-16T13:52:27.666 INFO:teuthology.orchestra.run.smithi124.stderr:0+1 records in 2024-01-16T13:52:27.666 INFO:teuthology.orchestra.run.smithi124.stderr:0+1 records out 2024-01-16T13:52:27.667 INFO:teuthology.orchestra.run.smithi124.stderr:68 bytes copied, 3.7481e-05 s, 1.8 MB/s 2024-01-16T13:52:27.668 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-16T13:52:27.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:28.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.073+0000 7f4a5e4e0700 1 Processor -- start 2024-01-16T13:52:28.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.073+0000 7f4a5e4e0700 1 -- start start 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a5e4e0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a580ce4d0 0x7f4a580d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a5e4e0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce910 0x7f4a580d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a5e4e0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a580d0150 con 0x7f4a5815e080 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a5e4e0700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5815e460 con 0x7f4a580ce4d0 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a5e4e0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580cc9b0 con 0x7f4a580ce910 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a577fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a580ce4d0 0x7f4a580d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a577fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4a580ce4d0 0x7f4a580d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41554/0 (socket says 172.21.15.113:41554) 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a577fe700 1 -- 172.21.15.113:0/860961200 learned_addr learned my addr 172.21.15.113:0/860961200 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:28.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.074+0000 7f4a5ca7d700 1 --2- 172.21.15.113:0/860961200 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce910 0x7f4a580d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a577fe700 1 -- 172.21.15.113:0/860961200 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce910 msgr2=0x7f4a580d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a577fe700 1 --2- 172.21.15.113:0/860961200 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce910 0x7f4a580d2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a577fe700 1 -- 172.21.15.113:0/860961200 >> v1:172.21.15.124:6789/0 conn(0x7f4a5815e080 legacy=0x7f4a580cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a577fe700 1 -- 172.21.15.113:0/860961200 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580cee50 con 0x7f4a580ce4d0 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a577fe700 1 --2- 172.21.15.113:0/860961200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a580ce4d0 0x7f4a580d1be0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009920 tx=0x7f4a4c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c5beb4fbc11de6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a56ffd700 1 -- 172.21.15.113:0/860961200 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c00ba20 con 0x7f4a580ce4d0 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a56ffd700 1 -- 172.21.15.113:0/860961200 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c00bb80 con 0x7f4a580ce4d0 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.075+0000 7f4a56ffd700 1 -- 172.21.15.113:0/860961200 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c009e70 con 0x7f4a580ce4d0 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/860961200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a580ce4d0 msgr2=0x7f4a580d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/860961200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a580ce4d0 0x7f4a580d1be0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009920 tx=0x7f4a4c009c30 comp rx=0 tx=0).stop 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/860961200 shutdown_connections 2024-01-16T13:52:28.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/860961200 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce910 0x7f4a580d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/860961200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a580ce4d0 0x7f4a580d1be0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/860961200 >> 172.21.15.113:0/860961200 conn(0x7f4a58073130 msgr2=0x7f4a5809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:28.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/860961200 shutdown_connections 2024-01-16T13:52:28.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.076+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/860961200 wait complete. 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.077+0000 7f4a5e4e0700 1 Processor -- start 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.077+0000 7f4a5e4e0700 1 -- start start 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5e4e0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce4d0 0x7f4a580c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5e4e0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4a580ce910 0x7f4a58159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5e4e0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 0x7f4a58159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5e4e0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a58157980 con 0x7f4a5815e080 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5e4e0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a581f3ca0 con 0x7f4a580ce910 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5e4e0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5815a880 con 0x7f4a580ce4d0 2024-01-16T13:52:28.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 0x7f4a58159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 0x7f4a58159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41568/0 (socket says 172.21.15.113:41568) 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 -- 172.21.15.113:0/786373751 learned_addr learned my addr 172.21.15.113:0/786373751 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a57fff700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce4d0 0x7f4a580c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a577fe700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4a580ce910 0x7f4a58159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 -- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce4d0 msgr2=0x7f4a580c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce4d0 0x7f4a580c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 -- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4a580ce910 msgr2=0x7f4a58159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4a580ce910 0x7f4a58159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 -- 172.21.15.113:0/786373751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c009550 con 0x7f4a5815e080 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.078+0000 7f4a5ca7d700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 0x7f4a58159ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a50007e30 tx=0x7f4a5000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.079+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5000eea0 con 0x7f4a5815e080 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.079+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50014420 con 0x7f4a5815e080 2024-01-16T13:52:28.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.079+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5815ab60 con 0x7f4a5815e080 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.079+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a500147d0 con 0x7f4a5815e080 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.079+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a58157eb0 con 0x7f4a5815e080 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.080+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a58067f60 con 0x7f4a5815e080 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.080+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4a50014930 con 0x7f4a5815e080 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.080+0000 7f4a54ff9700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4a4406b350 0x7f4a4406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.080+0000 7f4a57fff700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4a4406b350 0x7f4a4406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.080+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a5008a2b0 con 0x7f4a5815e080 2024-01-16T13:52:28.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.081+0000 7f4a57fff700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4a4406b350 0x7f4a4406d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4a581f72f0 tx=0x7f4a48009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.085+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a500551b0 con 0x7f4a5815e080 2024-01-16T13:52:28.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.272+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a580b7bd0 con 0x7f4a5815e080 2024-01-16T13:52:28.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.273+0000 7f4a54ff9700 1 -- 172.21.15.113:0/786373751 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f4a500587d0 con 0x7f4a5815e080 2024-01-16T13:52:28.256 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:28.256 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:28.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4a4406b350 msgr2=0x7f4a4406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4a4406b350 0x7f4a4406d810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4a581f72f0 tx=0x7f4a48009040 comp rx=0 tx=0).stop 2024-01-16T13:52:28.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 msgr2=0x7f4a58159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 0x7f4a58159ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a50007e30 tx=0x7f4a5000e3c0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 shutdown_connections 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4a4406b350 0x7f4a4406d810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a5815e080 0x7f4a58159ea0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4a580ce4d0 0x7f4a580c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 --2- 172.21.15.113:0/786373751 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4a580ce910 0x7f4a58159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 >> 172.21.15.113:0/786373751 conn(0x7f4a58073130 msgr2=0x7f4a5806ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 shutdown_connections 2024-01-16T13:52:28.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.275+0000 7f4a5e4e0700 1 -- 172.21.15.113:0/786373751 wait complete. 2024-01-16T13:52:28.271 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:28.271 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:28.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.413+0000 7f9c4fbf0700 1 Processor -- start 2024-01-16T13:52:28.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.413+0000 7f9c4fbf0700 1 -- start start 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4fbf0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c480cffc0 0x7f9c480b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4fbf0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c480d03a0 0x7f9c4815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4fbf0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480b3860 con 0x7f9c4815f700 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4fbf0700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4815fb90 con 0x7f9c480cffc0 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4fbf0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480b2520 con 0x7f9c480d03a0 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d98c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c480cffc0 0x7f9c480b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4e18d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f9c4815f700 0x7f9c4809aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55984/0 (socket says 172.21.15.113:55984) 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4e18d700 1 -- 172.21.15.113:0/2408381012 learned_addr learned my addr 172.21.15.113:0/2408381012 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d18b700 1 --2- 172.21.15.113:0/2408381012 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c480d03a0 0x7f9c4815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d98c700 1 -- 172.21.15.113:0/2408381012 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c480d03a0 msgr2=0x7f9c4815f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:28.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d98c700 1 --2- 172.21.15.113:0/2408381012 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c480d03a0 0x7f9c4815f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d98c700 1 -- 172.21.15.113:0/2408381012 >> v1:172.21.15.113:6789/0 conn(0x7f9c4815f700 legacy=0x7f9c4809aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4c98a700 1 -- 172.21.15.113:0/2408381012 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887769590 0 0) 0x7f9c40001490 con 0x7f9c4815f700 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d98c700 1 -- 172.21.15.113:0/2408381012 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480b2800 con 0x7f9c480cffc0 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4d98c700 1 --2- 172.21.15.113:0/2408381012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480cffc0 0x7f9c480b3320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009840 tx=0x7f9c44009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f839afab45b0899 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.414+0000 7f9c4c98a700 1 -- 172.21.15.113:0/2408381012 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44011830 con 0x7f9c480cffc0 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4c98a700 1 -- 172.21.15.113:0/2408381012 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c44011990 con 0x7f9c480cffc0 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4c98a700 1 -- 172.21.15.113:0/2408381012 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44011d20 con 0x7f9c480cffc0 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2408381012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480cffc0 msgr2=0x7f9c480b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2408381012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480cffc0 0x7f9c480b3320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009840 tx=0x7f9c44009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2408381012 shutdown_connections 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2408381012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480cffc0 0x7f9c480b3320 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2408381012 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c480d03a0 0x7f9c4815f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2408381012 >> 172.21.15.113:0/2408381012 conn(0x7f9c480d1e50 msgr2=0x7f9c480d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:28.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2408381012 shutdown_connections 2024-01-16T13:52:28.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.415+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2408381012 wait complete. 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.417+0000 7f9c4fbf0700 1 Processor -- start 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.417+0000 7f9c4fbf0700 1 -- start start 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c480cffc0 0x7f9c480b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 0x7f9c480c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c4815f700 0x7f9c481f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480cac10 con 0x7f9c480d03a0 2024-01-16T13:52:28.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480ca000 con 0x7f9c480cffc0 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480ca2e0 con 0x7f9c4815f700 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 0x7f9c480c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 0x7f9c480c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41588/0 (socket says 172.21.15.113:41588) 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 -- 172.21.15.113:0/2210726061 learned_addr learned my addr 172.21.15.113:0/2210726061 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4e18d700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c4815f700 0x7f9c481f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d98c700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c480cffc0 0x7f9c480b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 -- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c4815f700 msgr2=0x7f9c481f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c4815f700 0x7f9c481f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 -- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c480cffc0 msgr2=0x7f9c480b6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c480cffc0 0x7f9c480b6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 -- 172.21.15.113:0/2210726061 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c440094a0 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4e18d700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c4815f700 0x7f9c481f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4d18b700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 0x7f9c480c9950 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800e9b0 tx=0x7f9c3800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38009710 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3800c850 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3800cbc0 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c480ca5c0 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.418+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c480cbc50 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.419+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c48069020 con 0x7f9c480d03a0 2024-01-16T13:52:28.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.420+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9c3800cd20 con 0x7f9c480d03a0 2024-01-16T13:52:28.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.420+0000 7f9c3e7fc700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c2c06b350 0x7f9c2c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.420+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c38089cf0 con 0x7f9c480d03a0 2024-01-16T13:52:28.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.420+0000 7f9c4d98c700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c2c06b350 0x7f9c2c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.421+0000 7f9c4d98c700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c2c06b350 0x7f9c2c06d810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c440090b0 tx=0x7f9c4401c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.425+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c3808e070 con 0x7f9c480d03a0 2024-01-16T13:52:28.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.617+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c480d2700 con 0x7f9c480d03a0 2024-01-16T13:52:28.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.617+0000 7f9c3e7fc700 1 -- 172.21.15.113:0/2210726061 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c380210a0 con 0x7f9c480d03a0 2024-01-16T13:52:28.600 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c2c06b350 msgr2=0x7f9c2c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c2c06b350 0x7f9c2c06d810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c440090b0 tx=0x7f9c4401c040 comp rx=0 tx=0).stop 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 msgr2=0x7f9c480c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 0x7f9c480c9950 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800e9b0 tx=0x7f9c3800ed70 comp rx=0 tx=0).stop 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 shutdown_connections 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9c2c06b350 0x7f9c2c06d810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9c4815f700 0x7f9c481f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c480cffc0 0x7f9c480b6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 --2- 172.21.15.113:0/2210726061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c480d03a0 0x7f9c480c9950 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 >> 172.21.15.113:0/2210726061 conn(0x7f9c480d1e50 msgr2=0x7f9c480ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:28.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 shutdown_connections 2024-01-16T13:52:28.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.620+0000 7f9c4fbf0700 1 -- 172.21.15.113:0/2210726061 wait complete. 2024-01-16T13:52:28.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:28.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.755+0000 7fb201682700 1 Processor -- start 2024-01-16T13:52:28.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.756+0000 7fb201682700 1 -- start start 2024-01-16T13:52:28.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.756+0000 7fb201682700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1fc0afb80 0x7fb1fc073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.756+0000 7fb201682700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc15e4a0 con 0x7fb1fc0744d0 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.756+0000 7fb201682700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc06aff0 con 0x7fb1fc15e0c0 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.756+0000 7fb201682700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc0b0010 con 0x7fb1fc0afb80 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.756+0000 7fb1fb7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb1fc0744d0 0x7fb1fc0add40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55986/0 (socket says 172.21.15.113:55986) 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/3933315079 learned_addr learned my addr 172.21.15.113:0/3933315079 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fa7fc700 1 --2- 172.21.15.113:0/3933315079 >> v2:172.21.15.124:3300/0 conn(0x7fb1fc0afb80 0x7fb1fc073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220167386 0 0) 0x7fb1fc15e4a0 con 0x7fb1fc0744d0 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e40035b0 con 0x7fb1fc0744d0 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248124684 0 0) 0x7fb1e40035b0 con 0x7fb1fc0744d0 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 >> v2:172.21.15.124:3300/0 conn(0x7fb1fc0afb80 msgr2=0x7fb1fc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 --2- 172.21.15.113:0/3933315079 >> v2:172.21.15.124:3300/0 conn(0x7fb1fc0afb80 0x7fb1fc073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 >> v1:172.21.15.124:6789/0 conn(0x7fb1fc15e0c0 legacy=0x7fb1fc0b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc06b2d0 con 0x7fb1fc0744d0 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fb1f40019f0 con 0x7fb1fc0744d0 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 >> v1:172.21.15.113:6789/0 conn(0x7fb1fc0744d0 legacy=0x7fb1fc0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1e4003a80 0x7fb1e4003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4004850 0x7fb1e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e4005210 0x7fb1e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0b0010 con 0x7fb1e4005210 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e400c140 con 0x7fb1e4003a80 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e400c300 con 0x7fb1e4004850 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e4005210 0x7fb1e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1faffd700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1e4003a80 0x7fb1e4003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fa7fc700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4004850 0x7fb1e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4004850 msgr2=0x7fb1e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:28.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4004850 0x7fb1e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1e4003a80 msgr2=0x7fb1e4003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1e4003a80 0x7fb1e4003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.757+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/3933315079 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40035b0 con 0x7fb1e4005210 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/3933315079 reap_dead start 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e4005210 0x7fb1e400ba10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400a760 tx=0x7fb1e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a59be22ea45a1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400a250 con 0x7fb1e4005210 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f40072d0 con 0x7fb1e4005210 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb1f9ffb700 1 -- 172.21.15.113:0/3933315079 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4007680 con 0x7fb1e4005210 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 -- 172.21.15.113:0/3933315079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e4005210 msgr2=0x7fb1e400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e4005210 0x7fb1e400ba10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400a760 tx=0x7fb1e4005880 comp rx=0 tx=0).stop 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 -- 172.21.15.113:0/3933315079 shutdown_connections 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 --2- 172.21.15.113:0/3933315079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e4005210 0x7fb1e400ba10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 -- 172.21.15.113:0/3933315079 >> 172.21.15.113:0/3933315079 conn(0x7fb1fc0d0c00 msgr2=0x7fb1fc0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:28.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 -- 172.21.15.113:0/3933315079 shutdown_connections 2024-01-16T13:52:28.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.758+0000 7fb201682700 1 -- 172.21.15.113:0/3933315079 wait complete. 2024-01-16T13:52:28.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 Processor -- start 2024-01-16T13:52:28.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 -- start start 2024-01-16T13:52:28.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4005210 0x7fb1fc1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1fc15b0d0 0x7fb1fc1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 0x7fb1fc15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0ceb80 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0cdfb0 con 0x7fb1fc15b0d0 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb201682700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0ce250 con 0x7fb1e4005210 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.760+0000 7fb1fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 0x7fb1fc15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 0x7fb1fc15c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41594/0 (socket says 172.21.15.113:41594) 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/1111397946 learned_addr learned my addr 172.21.15.113:0/1111397946 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1faffd700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4005210 0x7fb1fc1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fa7fc700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1fc15b0d0 0x7fb1fc1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4005210 msgr2=0x7fb1fc1f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4005210 0x7fb1fc1f3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1fc15b0d0 msgr2=0x7fb1fc1582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1fc15b0d0 0x7fb1fc1582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 -- 172.21.15.113:0/1111397946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0ce530 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1faffd700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4005210 0x7fb1fc1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fa7fc700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1fc15b0d0 0x7fb1fc1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:28.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1fb7fe700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 0x7fb1fc15c9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0d0b90 tx=0x7fb1f4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4024030 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f4001250 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc0ce810 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f401f650 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.761+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc15cee0 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.762+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc15b4b0 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.762+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb1f4027070 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.763+0000 7fb1e37fe700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb1e806b380 0x7fb1e806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.763+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1f4025070 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.763+0000 7fb1faffd700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb1e806b380 0x7fb1e806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:28.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.763+0000 7fb1faffd700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb1e806b380 0x7fb1e806d840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0007fd0 tx=0x7fb1f0009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:28.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.767+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1fc15b4b0 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.961+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1fc15b4b0 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.962+0000 7fb1e37fe700 1 -- 172.21.15.113:0/1111397946 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1fc15b4b0 con 0x7fb1fc14d6e0 2024-01-16T13:52:28.945 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:28.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb1e806b380 msgr2=0x7fb1e806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb1e806b380 0x7fb1e806d840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0007fd0 tx=0x7fb1f0009040 comp rx=0 tx=0).stop 2024-01-16T13:52:28.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 msgr2=0x7fb1fc15c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:28.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 0x7fb1fc15c9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0d0b90 tx=0x7fb1f4001cb0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 shutdown_connections 2024-01-16T13:52:28.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb1e806b380 0x7fb1e806d840 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1fc14d6e0 0x7fb1fc15c9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1e4005210 0x7fb1fc1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 --2- 172.21.15.113:0/1111397946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1fc15b0d0 0x7fb1fc1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:28.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 >> 172.21.15.113:0/1111397946 conn(0x7fb1fc0d0c00 msgr2=0x7fb1fc0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:28.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 shutdown_connections 2024-01-16T13:52:28.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:28.964+0000 7fb201682700 1 -- 172.21.15.113:0/1111397946 wait complete. 2024-01-16T13:52:28.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:29.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.101+0000 7f7801227700 1 Processor -- start 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.101+0000 7f7801227700 1 -- start start 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f7801227700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 0x7f77fc0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f7801227700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f77fc0d2870 0x7f77fc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f7801227700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc0d23e0 con 0x7f77fc0d4590 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f7801227700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc0d5010 con 0x7f77fc0d2870 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f7801227700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc15e4f0 con 0x7f77fc15e110 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f77fa59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f77fc0d4590 0x7f77fc0b7000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55998/0 (socket says 172.21.15.113:55998) 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.102+0000 7f77fa59c700 1 -- 172.21.15.113:0/1946866334 learned_addr learned my addr 172.21.15.113:0/1946866334 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 0x7f77fc0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fb59e700 1 --2- 172.21.15.113:0/1946866334 >> v2:172.21.15.124:3300/0 conn(0x7f77fc0d2870 0x7f77fc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96974063 0 0) 0x7f77fc0d23e0 con 0x7f77fc0d4590 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e40035b0 con 0x7f77fc0d4590 2024-01-16T13:52:29.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977502586 0 0) 0x7f77e40035b0 con 0x7f77fc0d4590 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 msgr2=0x7f77fc0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 0x7f77fc0cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 >> v2:172.21.15.124:3300/0 conn(0x7f77fc0d2870 msgr2=0x7f77fc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 --2- 172.21.15.113:0/1946866334 >> v2:172.21.15.124:3300/0 conn(0x7f77fc0d2870 0x7f77fc0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc0d4d30 con 0x7f77fc0d4590 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f77ec001670 con 0x7f77fc0d4590 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 >> v1:172.21.15.113:6789/0 conn(0x7f77fc0d4590 legacy=0x7f77fc0b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4003a00 0x7f77e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 0x7f77e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77e4004d90 0x7f77e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc0d5010 con 0x7f77e4003a00 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc15e4f0 con 0x7f77e4004d90 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e400ba10 con 0x7f77fc15e110 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4003a00 0x7f77e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fa59c700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 0x7f77e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fb59e700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77e4004d90 0x7f77e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 -- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 msgr2=0x7f77e4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77fc15e110 0x7f77e4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 -- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77e4004d90 msgr2=0x7f77e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77e4004d90 0x7f77e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 -- 172.21.15.113:0/1946866334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40035b0 con 0x7f77e4003a00 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.103+0000 7f77fad9d700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4003a00 0x7f77e4003e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009b60 tx=0x7f77e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cea398cd72fa95 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f000e6c0 con 0x7f77e4003a00 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f000e820 con 0x7f77e4003a00 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f77f9d9b700 1 -- 172.21.15.113:0/1946866334 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f000eb90 con 0x7f77e4003a00 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f7801227700 1 -- 172.21.15.113:0/1946866334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4003a00 msgr2=0x7f77e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f7801227700 1 --2- 172.21.15.113:0/1946866334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77e4003a00 0x7f77e4003e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009b60 tx=0x7f77e4005880 comp rx=0 tx=0).stop 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f77fb59e700 1 -- 172.21.15.113:0/1946866334 reap_dead start 2024-01-16T13:52:29.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f7801227700 1 -- 172.21.15.113:0/1946866334 shutdown_connections 2024-01-16T13:52:29.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f7801227700 1 -- 172.21.15.113:0/1946866334 >> 172.21.15.113:0/1946866334 conn(0x7f77fc071a30 msgr2=0x7f77fc071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:29.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f7801227700 1 -- 172.21.15.113:0/1946866334 shutdown_connections 2024-01-16T13:52:29.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.104+0000 7f7801227700 1 -- 172.21.15.113:0/1946866334 wait complete. 2024-01-16T13:52:29.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 Processor -- start 2024-01-16T13:52:29.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 -- start start 2024-01-16T13:52:29.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77e4003a00 0x7f77fc1601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77fc160b40 0x7f77fc160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 0x7f77fc1f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc161500 con 0x7f77fc1f7420 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc154800 con 0x7f77fc160b40 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.106+0000 7f7801227700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1606f0 con 0x7f77e4003a00 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 0x7f77fc1f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 0x7f77fc1f78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41606/0 (socket says 172.21.15.113:41606) 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 -- 172.21.15.113:0/608802855 learned_addr learned my addr 172.21.15.113:0/608802855 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fad9d700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77e4003a00 0x7f77fc1601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fa59c700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77fc160b40 0x7f77fc160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 -- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77e4003a00 msgr2=0x7f77fc1601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77e4003a00 0x7f77fc1601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 -- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77fc160b40 msgr2=0x7f77fc160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77fc160b40 0x7f77fc160fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 -- 172.21.15.113:0/608802855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc15f4b0 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77fb59e700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 0x7f77fc1f78a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f77fc06d1d0 tx=0x7f77f4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f400e040 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f400f040 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f400fc40 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc15f790 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.107+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc15fd50 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.108+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc0b3c10 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.108+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f77f4022460 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.109+0000 7f77e37fe700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f77e806b350 0x7f77e806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.109+0000 7f77fad9d700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f77e806b350 0x7f77e806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.109+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77f4089f30 con 0x7f77fc1f7420 2024-01-16T13:52:29.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.109+0000 7f77fad9d700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f77e806b350 0x7f77e806d810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f77f00090b0 tx=0x7f77f0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.113+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77fc0b3c10 con 0x7f77fc1f7420 2024-01-16T13:52:29.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.303+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f77fc0b7690 con 0x7f77fc1f7420 2024-01-16T13:52:29.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.303+0000 7f77e37fe700 1 -- 172.21.15.113:0/608802855 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f77f4058830 con 0x7f77fc1f7420 2024-01-16T13:52:29.286 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:29.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f77e806b350 msgr2=0x7f77e806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f77e806b350 0x7f77e806d810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f77f00090b0 tx=0x7f77f0008030 comp rx=0 tx=0).stop 2024-01-16T13:52:29.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 msgr2=0x7f77fc1f78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 0x7f77fc1f78a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f77fc06d1d0 tx=0x7f77f4011ed0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 shutdown_connections 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f77e806b350 0x7f77e806d810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc1f7420 0x7f77fc1f78a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f77e4003a00 0x7f77fc1601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 --2- 172.21.15.113:0/608802855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f77fc160b40 0x7f77fc160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 >> 172.21.15.113:0/608802855 conn(0x7f77fc071a30 msgr2=0x7f77fc0b9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 shutdown_connections 2024-01-16T13:52:29.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.305+0000 7f7801227700 1 -- 172.21.15.113:0/608802855 wait complete. 2024-01-16T13:52:29.300 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:29.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.439+0000 7fef31ec1700 1 Processor -- start 2024-01-16T13:52:29.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef31ec1700 1 -- start start 2024-01-16T13:52:29.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef31ec1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b4350 0x7fef2c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef31ec1700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c0b0290 con 0x7fef2c0b5dd0 2024-01-16T13:52:29.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef31ec1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c0b4820 con 0x7fef2c0b59f0 2024-01-16T13:52:29.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef31ec1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0b4b00 con 0x7fef2c0b4350 2024-01-16T13:52:29.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef2affd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fef2c0b5dd0 0x7fef2c0b3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56014/0 (socket says 172.21.15.113:56014) 2024-01-16T13:52:29.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.440+0000 7fef2affd700 1 -- 172.21.15.113:0/3222550598 learned_addr learned my addr 172.21.15.113:0/3222550598 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2bfff700 1 --2- 172.21.15.113:0/3222550598 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b4350 0x7fef2c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2a7fc700 1 -- 172.21.15.113:0/3222550598 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262061533 0 0) 0x7fef2c0b0290 con 0x7fef2c0b5dd0 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2a7fc700 1 -- 172.21.15.113:0/3222550598 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef180035b0 con 0x7fef2c0b5dd0 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2a7fc700 1 -- 172.21.15.113:0/3222550598 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96401523 0 0) 0x7fef2c0b4820 con 0x7fef2c0b59f0 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2bfff700 1 -- 172.21.15.113:0/3222550598 >> v1:172.21.15.124:6789/0 conn(0x7fef2c0b59f0 legacy=0x7fef2c06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2bfff700 1 -- 172.21.15.113:0/3222550598 >> v1:172.21.15.113:6789/0 conn(0x7fef2c0b5dd0 legacy=0x7fef2c0b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2bfff700 1 -- 172.21.15.113:0/3222550598 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0ae9b0 con 0x7fef2c0b4350 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2bfff700 1 --2- 172.21.15.113:0/3222550598 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b4350 0x7fef2c0af190 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fef24009df0 tx=0x7fef2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c35b3b9627fa7c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2a7fc700 1 -- 172.21.15.113:0/3222550598 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2400e070 con 0x7fef2c0b4350 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2a7fc700 1 -- 172.21.15.113:0/3222550598 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef24003a20 con 0x7fef2c0b4350 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.441+0000 7fef2a7fc700 1 -- 172.21.15.113:0/3222550598 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14001670 con 0x7fef2c0b4350 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 -- 172.21.15.113:0/3222550598 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b4350 msgr2=0x7fef2c0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 --2- 172.21.15.113:0/3222550598 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b4350 0x7fef2c0af190 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fef24009df0 tx=0x7fef2400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:29.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 -- 172.21.15.113:0/3222550598 shutdown_connections 2024-01-16T13:52:29.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 --2- 172.21.15.113:0/3222550598 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b4350 0x7fef2c0af190 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 -- 172.21.15.113:0/3222550598 >> 172.21.15.113:0/3222550598 conn(0x7fef2c072990 msgr2=0x7fef2c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:29.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 -- 172.21.15.113:0/3222550598 shutdown_connections 2024-01-16T13:52:29.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.442+0000 7fef31ec1700 1 -- 172.21.15.113:0/3222550598 wait complete. 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.443+0000 7fef31ec1700 1 Processor -- start 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.443+0000 7fef31ec1700 1 -- start start 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef31ec1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fef2c0b4350 0x7fef2c15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef31ec1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 0x7fef2c14a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef31ec1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b5dd0 0x7fef2c15dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef31ec1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c15e830 con 0x7fef2c0b59f0 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef31ec1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c157240 con 0x7fef2c0b4350 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef31ec1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0cd060 con 0x7fef2c0b5dd0 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef2affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 0x7fef2c14a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef2affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 0x7fef2c14a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41622/0 (socket says 172.21.15.113:41622) 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef2affd700 1 -- 172.21.15.113:0/1197237533 learned_addr learned my addr 172.21.15.113:0/1197237533 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef2b7fe700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fef2c0b4350 0x7fef2c15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef2affd700 1 -- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b5dd0 msgr2=0x7fef2c15dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.444+0000 7fef2affd700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b5dd0 0x7fef2c15dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef2affd700 1 -- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fef2c0b4350 msgr2=0x7fef2c15edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef2affd700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fef2c0b4350 0x7fef2c15edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef2affd700 1 -- 172.21.15.113:0/1197237533 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24009a70 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef2affd700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 0x7fef2c14a2a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef1400c760 tx=0x7fef1400cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14005d70 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef14007070 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c0cd340 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14007420 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.445+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c0cd900 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.446+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c067f60 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.446+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fef14007580 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.446+0000 7fef23fff700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fef0c06b350 0x7fef0c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.447+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef14088ff0 con 0x7fef2c0b59f0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.447+0000 7fef2b7fe700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fef0c06b350 0x7fef0c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.447+0000 7fef2b7fe700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fef0c06b350 0x7fef0c06d810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000f60 tx=0x7fef1c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.451+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef14053f10 con 0x7fef2c0b59f0 2024-01-16T13:52:29.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.644+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef2c0b0cc0 con 0x7fef2c0b59f0 2024-01-16T13:52:29.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.645+0000 7fef23fff700 1 -- 172.21.15.113:0/1197237533 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fef14017090 con 0x7fef2c0b59f0 2024-01-16T13:52:29.629 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:29.629 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.647+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fef0c06b350 msgr2=0x7fef0c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.647+0000 7fef31ec1700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fef0c06b350 0x7fef0c06d810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000f60 tx=0x7fef1c009040 comp rx=0 tx=0).stop 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.647+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 msgr2=0x7fef2c14a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.647+0000 7fef31ec1700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 0x7fef2c14a2a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef1400c760 tx=0x7fef1400cb20 comp rx=0 tx=0).stop 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 shutdown_connections 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fef0c06b350 0x7fef0c06d810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fef2c0b5dd0 0x7fef2c15dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fef2c0b4350 0x7fef2c15edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 --2- 172.21.15.113:0/1197237533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0b59f0 0x7fef2c14a2a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 >> 172.21.15.113:0/1197237533 conn(0x7fef2c072990 msgr2=0x7fef2c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:29.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 shutdown_connections 2024-01-16T13:52:29.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.648+0000 7fef31ec1700 1 -- 172.21.15.113:0/1197237533 wait complete. 2024-01-16T13:52:29.643 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:29.643 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:29.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.784+0000 7eff64a2f700 1 Processor -- start 2024-01-16T13:52:29.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.784+0000 7eff64a2f700 1 -- start start 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.785+0000 7eff64a2f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7eff6006db80 0x7eff60156400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.785+0000 7eff64a2f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6006f5e0 con 0x7eff6006c710 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.785+0000 7eff64a2f700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6006f8c0 con 0x7eff600cf740 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.785+0000 7eff64a2f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6006fba0 con 0x7eff6006db80 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5ed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7eff6006db80 0x7eff60156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5dd9b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7eff6006c710 0x7eff60155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56022/0 (socket says 172.21.15.113:56022) 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5dd9b700 1 -- 172.21.15.113:0/271858841 learned_addr learned my addr 172.21.15.113:0/271858841 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5ed9d700 1 -- 172.21.15.113:0/271858841 >> v1:172.21.15.124:6789/0 conn(0x7eff600cf740 legacy=0x7eff6006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5ed9d700 1 -- 172.21.15.113:0/271858841 >> v1:172.21.15.113:6789/0 conn(0x7eff6006c710 legacy=0x7eff60155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5ed9d700 1 -- 172.21.15.113:0/271858841 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff600728d0 con 0x7eff6006db80 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5d59a700 1 -- 172.21.15.113:0/271858841 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141535122 0 0) 0x7eff6006f5e0 con 0x7eff6006c710 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5ed9d700 1 --2- 172.21.15.113:0/271858841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006db80 0x7eff60156400 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7eff58009d90 tx=0x7eff5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8bde905d5309b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5d59a700 1 -- 172.21.15.113:0/271858841 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5800e070 con 0x7eff6006db80 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5d59a700 1 -- 172.21.15.113:0/271858841 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5800bbd0 con 0x7eff6006db80 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.786+0000 7eff5d59a700 1 -- 172.21.15.113:0/271858841 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58003a20 con 0x7eff6006db80 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 -- 172.21.15.113:0/271858841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006db80 msgr2=0x7eff60156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 --2- 172.21.15.113:0/271858841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006db80 0x7eff60156400 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7eff58009d90 tx=0x7eff5800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 -- 172.21.15.113:0/271858841 shutdown_connections 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 --2- 172.21.15.113:0/271858841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006db80 0x7eff60156400 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 -- 172.21.15.113:0/271858841 >> 172.21.15.113:0/271858841 conn(0x7eff600ce500 msgr2=0x7eff60071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 -- 172.21.15.113:0/271858841 shutdown_connections 2024-01-16T13:52:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.787+0000 7eff64a2f700 1 -- 172.21.15.113:0/271858841 wait complete. 2024-01-16T13:52:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.788+0000 7eff64a2f700 1 Processor -- start 2024-01-16T13:52:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.788+0000 7eff64a2f700 1 -- start start 2024-01-16T13:52:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff64a2f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 0x7eff60150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff64a2f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7eff6006db80 0x7eff6015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff64a2f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff600cf740 0x7eff60147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff64a2f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff60156940 con 0x7eff6006c710 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff64a2f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6015ebc0 con 0x7eff600cf740 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff64a2f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6014c2f0 con 0x7eff6006db80 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.789+0000 7eff5e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 0x7eff60150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 0x7eff60150a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41630/0 (socket says 172.21.15.113:41630) 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 -- 172.21.15.113:0/3364889561 learned_addr learned my addr 172.21.15.113:0/3364889561 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 -- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7eff6006db80 msgr2=0x7eff6015c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7eff6006db80 0x7eff6015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 -- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff600cf740 msgr2=0x7eff60147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5ed9d700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff600cf740 0x7eff60147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff600cf740 0x7eff60147910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 -- 172.21.15.113:0/3364889561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58009a70 con 0x7eff6006c710 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff5e59c700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 0x7eff60150a50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7eff5400c4d0 tx=0x7eff5400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff540054e0 con 0x7eff6006c710 2024-01-16T13:52:29.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff54003680 con 0x7eff6006c710 2024-01-16T13:52:29.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54003a30 con 0x7eff6006c710 2024-01-16T13:52:29.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6015d2e0 con 0x7eff6006c710 2024-01-16T13:52:29.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.790+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6015ca40 con 0x7eff6006c710 2024-01-16T13:52:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.791+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff60069020 con 0x7eff6006c710 2024-01-16T13:52:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.792+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff54003b90 con 0x7eff6006c710 2024-01-16T13:52:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.792+0000 7eff4effd700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff4006b380 0x7eff4006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.792+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff54088bc0 con 0x7eff6006c710 2024-01-16T13:52:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.792+0000 7eff5dd9b700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff4006b380 0x7eff4006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.793+0000 7eff5dd9b700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff4006b380 0x7eff4006d840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff60150240 tx=0x7eff500079f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:29.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.797+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff54053ac0 con 0x7eff6006c710 2024-01-16T13:52:29.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.986+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff600735c0 con 0x7eff6006c710 2024-01-16T13:52:29.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.987+0000 7eff4effd700 1 -- 172.21.15.113:0/3364889561 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff540570e0 con 0x7eff6006c710 2024-01-16T13:52:29.970 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:29.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff4006b380 msgr2=0x7eff4006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff4006b380 0x7eff4006d840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff60150240 tx=0x7eff500079f0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 msgr2=0x7eff60150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:29.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 0x7eff60150a50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7eff5400c4d0 tx=0x7eff5400c890 comp rx=0 tx=0).stop 2024-01-16T13:52:29.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 shutdown_connections 2024-01-16T13:52:29.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff4006b380 0x7eff4006d840 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff600cf740 0x7eff60147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff6006c710 0x7eff60150a50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 --2- 172.21.15.113:0/3364889561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7eff6006db80 0x7eff6015c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:29.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 >> 172.21.15.113:0/3364889561 conn(0x7eff600ce500 msgr2=0x7eff600ae230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:29.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 shutdown_connections 2024-01-16T13:52:29.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:29.989+0000 7eff64a2f700 1 -- 172.21.15.113:0/3364889561 wait complete. 2024-01-16T13:52:29.984 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.125+0000 7f9355428700 1 Processor -- start 2024-01-16T13:52:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.125+0000 7f9355428700 1 -- start start 2024-01-16T13:52:30.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f9355428700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9350161510 0x7f93500cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f9355428700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f93500cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f9355428700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f93500b59d0 0x7f93500b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f9355428700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9350162af0 con 0x7f9350161510 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f9355428700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500d13e0 con 0x7f93500b59d0 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f9355428700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9350161f70 con 0x7f93500b98e0 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f934effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9350161510 0x7f93500cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f934effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9350161510 0x7f93500cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41644/0 (socket says 172.21.15.113:41644) 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f934effd700 1 -- 172.21.15.113:0/2748688649 learned_addr learned my addr 172.21.15.113:0/2748688649 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f934e7fc700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f93500cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f934f7fe700 1 --2- 172.21.15.113:0/2748688649 >> v2:172.21.15.124:3300/0 conn(0x7f93500b59d0 0x7f93500b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.126+0000 7f934effd700 1 -- 172.21.15.113:0/2748688649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 msgr2=0x7f93500cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934effd700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f93500cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934effd700 1 -- 172.21.15.113:0/2748688649 >> v2:172.21.15.124:3300/0 conn(0x7f93500b59d0 msgr2=0x7f93500b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934effd700 1 --2- 172.21.15.113:0/2748688649 >> v2:172.21.15.124:3300/0 conn(0x7f93500b59d0 0x7f93500b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934effd700 1 -- 172.21.15.113:0/2748688649 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350162250 con 0x7f9350161510 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934f7fe700 1 --2- 172.21.15.113:0/2748688649 >> v2:172.21.15.124:3300/0 conn(0x7f93500b59d0 0x7f93500b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:30.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934e7fc700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f93500cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934effd700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cec20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f93500b9d90 tx=0x7f9344009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a07168c46d0068 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934dffb700 1 -- 172.21.15.113:0/2748688649 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934400b870 con 0x7f9350161510 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934dffb700 1 -- 172.21.15.113:0/2748688649 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934400b9d0 con 0x7f9350161510 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f934dffb700 1 -- 172.21.15.113:0/2748688649 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934400bd60 con 0x7f9350161510 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 -- 172.21.15.113:0/2748688649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 msgr2=0x7f93500cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cec20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f93500b9d90 tx=0x7f9344009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 -- 172.21.15.113:0/2748688649 shutdown_connections 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cec20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 --2- 172.21.15.113:0/2748688649 >> v2:172.21.15.124:3300/0 conn(0x7f93500b59d0 0x7f93500b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 --2- 172.21.15.113:0/2748688649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f93500cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.127+0000 7f9355428700 1 -- 172.21.15.113:0/2748688649 >> 172.21.15.113:0/2748688649 conn(0x7f935006b410 msgr2=0x7f935006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.128+0000 7f9355428700 1 -- 172.21.15.113:0/2748688649 shutdown_connections 2024-01-16T13:52:30.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.128+0000 7f9355428700 1 -- 172.21.15.113:0/2748688649 wait complete. 2024-01-16T13:52:30.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.129+0000 7f9355428700 1 Processor -- start 2024-01-16T13:52:30.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.129+0000 7f9355428700 1 -- start start 2024-01-16T13:52:30.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93500b59d0 0x7f93500c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f9350160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500cc520 con 0x7f9350161510 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500cc770 con 0x7f93500b59d0 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93500cca50 con 0x7f93500b98e0 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41658/0 (socket says 172.21.15.113:41658) 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 -- 172.21.15.113:0/1996766919 learned_addr learned my addr 172.21.15.113:0/1996766919 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934e7fc700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f9350160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934effd700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93500b59d0 0x7f93500c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 -- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 msgr2=0x7f9350160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f9350160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 -- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93500b59d0 msgr2=0x7f93500c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93500b59d0 0x7f93500c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 -- 172.21.15.113:0/1996766919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9344009550 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f934f7fe700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cd810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f934800cef0 tx=0x7f934800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934800ac80 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934800ade0 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348015680 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93500ccd30 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.130+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93500cd2f0 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.131+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9350068450 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.132+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f934800d430 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.132+0000 7f93377fe700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f933c06b350 0x7f933c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.132+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9348089f30 con 0x7f9350161510 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.132+0000 7f934effd700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f933c06b350 0x7f933c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.133+0000 7f934effd700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f933c06b350 0x7f933c06d810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9344014040 tx=0x7f9344008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.137+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9348054e30 con 0x7f9350161510 2024-01-16T13:52:30.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.327+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f93500cd510 con 0x7f9350161510 2024-01-16T13:52:30.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.327+0000 7f93377fe700 1 -- 172.21.15.113:0/1996766919 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9348058450 con 0x7f9350161510 2024-01-16T13:52:30.311 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f933c06b350 msgr2=0x7f933c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f933c06b350 0x7f933c06d810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9344014040 tx=0x7f9344008030 comp rx=0 tx=0).stop 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 msgr2=0x7f93500cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cd810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f934800cef0 tx=0x7f934800cf20 comp rx=0 tx=0).stop 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 shutdown_connections 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f933c06b350 0x7f933c06d810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9350161510 0x7f93500cd810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93500b59d0 0x7f93500c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 --2- 172.21.15.113:0/1996766919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93500b98e0 0x7f9350160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.329+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 >> 172.21.15.113:0/1996766919 conn(0x7f935006b410 msgr2=0x7f93500b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:30.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.330+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 shutdown_connections 2024-01-16T13:52:30.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.330+0000 7f9355428700 1 -- 172.21.15.113:0/1996766919 wait complete. 2024-01-16T13:52:30.325 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:30.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.467+0000 7f764f662700 1 Processor -- start 2024-01-16T13:52:30.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.467+0000 7f764f662700 1 -- start start 2024-01-16T13:52:30.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764f662700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f76480d0080 0x7f764806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764f662700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76480d0460 0x7f76480d2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764f662700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7648070e90 con 0x7f764815faa0 2024-01-16T13:52:30.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764f662700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76480d1750 con 0x7f76480d0460 2024-01-16T13:52:30.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764f662700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76480d27a0 con 0x7f76480d0080 2024-01-16T13:52:30.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76480d0460 0x7f76480d2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76480d0460 0x7f76480d2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41668/0 (socket says 172.21.15.113:41668) 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 -- 172.21.15.113:0/1514067911 learned_addr learned my addr 172.21.15.113:0/1514067911 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764d3fe700 1 --2- 172.21.15.113:0/1514067911 >> v2:172.21.15.124:3300/0 conn(0x7f76480d0080 0x7f764806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 -- 172.21.15.113:0/1514067911 >> v2:172.21.15.124:3300/0 conn(0x7f76480d0080 msgr2=0x7f764806ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 --2- 172.21.15.113:0/1514067911 >> v2:172.21.15.124:3300/0 conn(0x7f76480d0080 0x7f764806ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 -- 172.21.15.113:0/1514067911 >> v1:172.21.15.124:6789/0 conn(0x7f764815faa0 legacy=0x7f764815fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 -- 172.21.15.113:0/1514067911 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480705f0 con 0x7f76480d0460 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f764cbfd700 1 --2- 172.21.15.113:0/1514067911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0460 0x7f76480d2260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f764806eb20 tx=0x7f7638009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c794e885640ae44d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.468+0000 7f763ffff700 1 -- 172.21.15.113:0/1514067911 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638004230 con 0x7f76480d0460 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f763ffff700 1 -- 172.21.15.113:0/1514067911 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7638004390 con 0x7f76480d0460 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f763ffff700 1 -- 172.21.15.113:0/1514067911 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380036a0 con 0x7f76480d0460 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 -- 172.21.15.113:0/1514067911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0460 msgr2=0x7f76480d2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 --2- 172.21.15.113:0/1514067911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0460 0x7f76480d2260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f764806eb20 tx=0x7f7638009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 -- 172.21.15.113:0/1514067911 shutdown_connections 2024-01-16T13:52:30.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 --2- 172.21.15.113:0/1514067911 >> v2:172.21.15.124:3300/0 conn(0x7f76480d0080 0x7f764806ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 --2- 172.21.15.113:0/1514067911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0460 0x7f76480d2260 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 -- 172.21.15.113:0/1514067911 >> 172.21.15.113:0/1514067911 conn(0x7f764809aa00 msgr2=0x7f764809ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:30.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 -- 172.21.15.113:0/1514067911 shutdown_connections 2024-01-16T13:52:30.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.469+0000 7f764f662700 1 -- 172.21.15.113:0/1514067911 wait complete. 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.470+0000 7f764f662700 1 Processor -- start 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 -- start start 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 0x7f764814efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76480d0460 0x7f76480cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f764815faa0 0x7f76480ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764806d000 con 0x7f76480d0080 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764814f4e0 con 0x7f76480d0460 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764f662700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764806b120 con 0x7f764815faa0 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764d3fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 0x7f764814efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764d3fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 0x7f764814efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41672/0 (socket says 172.21.15.113:41672) 2024-01-16T13:52:30.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.471+0000 7f764d3fe700 1 -- 172.21.15.113:0/4016249104 learned_addr learned my addr 172.21.15.113:0/4016249104 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764cbfd700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76480d0460 0x7f76480cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764d3fe700 1 -- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f764815faa0 msgr2=0x7f76480ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764dbff700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f764815faa0 0x7f76480ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764d3fe700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f764815faa0 0x7f76480ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764d3fe700 1 -- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76480d0460 msgr2=0x7f76480cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764d3fe700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76480d0460 0x7f76480cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764d3fe700 1 -- 172.21.15.113:0/4016249104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638009550 con 0x7f76480d0080 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764cbfd700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76480d0460 0x7f76480cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764dbff700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f764815faa0 0x7f76480ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764d3fe700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 0x7f764814efa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f764400c7f0 tx=0x7f764400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644011800 con 0x7f76480d0080 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644011960 con 0x7f76480d0080 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764806b400 con 0x7f76480d0080 2024-01-16T13:52:30.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644011cf0 con 0x7f76480d0080 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.472+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764806b9c0 con 0x7f76480d0080 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.473+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648069020 con 0x7f76480d0080 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.474+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7644018360 con 0x7f76480d0080 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.474+0000 7f763dffb700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f762c06b350 0x7f762c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.474+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7644089a90 con 0x7f76480d0080 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.474+0000 7f764cbfd700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f762c06b350 0x7f762c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.474+0000 7f764cbfd700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f762c06b350 0x7f762c06d810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7648157880 tx=0x7f7638008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.478+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7644054990 con 0x7f76480d0080 2024-01-16T13:52:30.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.669+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f764806ee10 con 0x7f76480d0080 2024-01-16T13:52:30.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.670+0000 7f763dffb700 1 -- 172.21.15.113:0/4016249104 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f764401f0a0 con 0x7f76480d0080 2024-01-16T13:52:30.653 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f762c06b350 msgr2=0x7f762c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f762c06b350 0x7f762c06d810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7648157880 tx=0x7f7638008030 comp rx=0 tx=0).stop 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 msgr2=0x7f764814efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 0x7f764814efa0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f764400c7f0 tx=0x7f764400cbb0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 shutdown_connections 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f762c06b350 0x7f762c06d810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f764815faa0 0x7f76480ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76480d0080 0x7f764814efa0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 --2- 172.21.15.113:0/4016249104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76480d0460 0x7f76480cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 >> 172.21.15.113:0/4016249104 conn(0x7f764809aa00 msgr2=0x7f764815d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:30.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 shutdown_connections 2024-01-16T13:52:30.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.672+0000 7f764f662700 1 -- 172.21.15.113:0/4016249104 wait complete. 2024-01-16T13:52:30.667 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:30.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.805+0000 7f0b287ef700 1 Processor -- start 2024-01-16T13:52:30.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.805+0000 7f0b287ef700 1 -- start start 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.806+0000 7f0b287ef700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0b2015e080 0x7f0b200b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.806+0000 7f0b287ef700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0b2006c710 0x7f0b200afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.806+0000 7f0b287ef700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b200ae430 con 0x7f0b200af4d0 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.806+0000 7f0b287ef700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b200af0b0 con 0x7f0b2006c710 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.806+0000 7f0b287ef700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b200b05a0 con 0x7f0b2015e080 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0b2006c710 0x7f0b200afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0b2006c710 0x7f0b200afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41680/0 (socket says 172.21.15.113:41680) 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2161277191 learned_addr learned my addr 172.21.15.113:0/2161277191 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2161277191 >> v2:172.21.15.124:3300/0 conn(0x7f0b2015e080 msgr2=0x7f0b200b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 --2- 172.21.15.113:0/2161277191 >> v2:172.21.15.124:3300/0 conn(0x7f0b2015e080 0x7f0b200b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2161277191 >> v1:172.21.15.124:6789/0 conn(0x7f0b200af4d0 legacy=0x7f0b2006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2161277191 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200ae710 con 0x7f0b2006c710 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b26d8c700 1 --2- 172.21.15.113:0/2161277191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2006c710 0x7f0b200afb90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b20072a90 tx=0x7f0b1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6002b747a4943be5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b25589700 1 -- 172.21.15.113:0/2161277191 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800e070 con 0x7f0b2006c710 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b25589700 1 -- 172.21.15.113:0/2161277191 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18003a20 con 0x7f0b2006c710 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.807+0000 7f0b25589700 1 -- 172.21.15.113:0/2161277191 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18003dd0 con 0x7f0b2006c710 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 -- 172.21.15.113:0/2161277191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2006c710 msgr2=0x7f0b200afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2161277191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2006c710 0x7f0b200afb90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b20072a90 tx=0x7f0b1800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 -- 172.21.15.113:0/2161277191 shutdown_connections 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2161277191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2006c710 0x7f0b200afb90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2161277191 >> v2:172.21.15.124:3300/0 conn(0x7f0b2015e080 0x7f0b200b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 -- 172.21.15.113:0/2161277191 >> 172.21.15.113:0/2161277191 conn(0x7f0b20071b90 msgr2=0x7f0b20071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 -- 172.21.15.113:0/2161277191 shutdown_connections 2024-01-16T13:52:30.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.808+0000 7f0b287ef700 1 -- 172.21.15.113:0/2161277191 wait complete. 2024-01-16T13:52:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.809+0000 7f0b287ef700 1 Processor -- start 2024-01-16T13:52:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.809+0000 7f0b287ef700 1 -- start start 2024-01-16T13:52:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b2006c710 0x7f0b20141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b200af4d0 0x7f0b2015b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 0x7f0b2006b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b20156e50 con 0x7f0b2015e080 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b20156690 con 0x7f0b2006c710 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b20142360 con 0x7f0b200af4d0 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 0x7f0b2006b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 0x7f0b2006b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41696/0 (socket says 172.21.15.113:41696) 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2129263123 learned_addr learned my addr 172.21.15.113:0/2129263123 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b25d8a700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b200af4d0 0x7f0b2015b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b2658b700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b2006c710 0x7f0b20141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b200af4d0 msgr2=0x7f0b2015b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b200af4d0 0x7f0b2015b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b2006c710 msgr2=0x7f0b20141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b2006c710 0x7f0b20141e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 -- 172.21.15.113:0/2129263123 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18009b30 con 0x7f0b2015e080 2024-01-16T13:52:30.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b2658b700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b2006c710 0x7f0b20141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b26d8c700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 0x7f0b2006b5f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b18015040 tx=0x7f0b18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18004240 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b180043c0 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b201425e0 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18004770 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.810+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b20156890 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.811+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b200753f0 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.812+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0b18019070 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.812+0000 7f0b16ffd700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0b0406b380 0x7f0b0406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.812+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b1808ab60 con 0x7f0b2015e080 2024-01-16T13:52:30.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.812+0000 7f0b2658b700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0b0406b380 0x7f0b0406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:30.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.813+0000 7f0b2658b700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0b0406b380 0x7f0b0406d840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000a8f0 tx=0x7f0b10008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:30.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:30.817+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b180559d0 con 0x7f0b2015e080 2024-01-16T13:52:30.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.010+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b2006e590 con 0x7f0b2015e080 2024-01-16T13:52:30.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.011+0000 7f0b16ffd700 1 -- 172.21.15.113:0/2129263123 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f0b18020020 con 0x7f0b2015e080 2024-01-16T13:52:30.995 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:30.995 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0b0406b380 msgr2=0x7f0b0406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0b0406b380 0x7f0b0406d840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000a8f0 tx=0x7f0b10008040 comp rx=0 tx=0).stop 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 msgr2=0x7f0b2006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 0x7f0b2006b5f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b18015040 tx=0x7f0b18004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 shutdown_connections 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0b0406b380 0x7f0b0406d840 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b2015e080 0x7f0b2006b5f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b2006c710 0x7f0b20141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 --2- 172.21.15.113:0/2129263123 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b200af4d0 0x7f0b2015b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 >> 172.21.15.113:0/2129263123 conn(0x7f0b20071b90 msgr2=0x7f0b200b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 shutdown_connections 2024-01-16T13:52:30.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.013+0000 7f0b287ef700 1 -- 172.21.15.113:0/2129263123 wait complete. 2024-01-16T13:52:31.009 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:31.009 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:31.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.150+0000 7f31bff7a700 1 Processor -- start 2024-01-16T13:52:31.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.150+0000 7f31bff7a700 1 -- start start 2024-01-16T13:52:31.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.150+0000 7f31bff7a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f31b81551c0 0x7f31b80ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bff7a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b815b390 0x7f31b80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bff7a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31b80cee20 0x7f31b80b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bff7a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b80cf330 con 0x7f31b80cee20 2024-01-16T13:52:31.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bff7a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b80cf610 con 0x7f31b81551c0 2024-01-16T13:52:31.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bff7a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80cf8f0 con 0x7f31b815b390 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31b80cee20 0x7f31b80b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31b80cee20 0x7f31b80b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41698/0 (socket says 172.21.15.113:41698) 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 -- 172.21.15.113:0/3959790724 learned_addr learned my addr 172.21.15.113:0/3959790724 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bd515700 1 --2- 172.21.15.113:0/3959790724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b815b390 0x7f31b80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bdd16700 1 --2- 172.21.15.113:0/3959790724 >> v2:172.21.15.124:3300/0 conn(0x7f31b81551c0 0x7f31b80ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 -- 172.21.15.113:0/3959790724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b815b390 msgr2=0x7f31b80b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 --2- 172.21.15.113:0/3959790724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b815b390 0x7f31b80b4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 -- 172.21.15.113:0/3959790724 >> v2:172.21.15.124:3300/0 conn(0x7f31b81551c0 msgr2=0x7f31b80ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 --2- 172.21.15.113:0/3959790724 >> v2:172.21.15.124:3300/0 conn(0x7f31b81551c0 0x7f31b80ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 -- 172.21.15.113:0/3959790724 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80cfbd0 con 0x7f31b80cee20 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bdd16700 1 --2- 172.21.15.113:0/3959790724 >> v2:172.21.15.124:3300/0 conn(0x7f31b81551c0 0x7f31b80ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31be517700 1 --2- 172.21.15.113:0/3959790724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b80cee20 0x7f31b80b5300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f31b80725a0 tx=0x7f31b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74db89dd1f03b0fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bcd14700 1 -- 172.21.15.113:0/3959790724 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b000e070 con 0x7f31b80cee20 2024-01-16T13:52:31.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bcd14700 1 -- 172.21.15.113:0/3959790724 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b000b9c0 con 0x7f31b80cee20 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.151+0000 7f31bcd14700 1 -- 172.21.15.113:0/3959790724 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b000bd70 con 0x7f31b80cee20 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 -- 172.21.15.113:0/3959790724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b80cee20 msgr2=0x7f31b80b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 --2- 172.21.15.113:0/3959790724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b80cee20 0x7f31b80b5300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f31b80725a0 tx=0x7f31b000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 -- 172.21.15.113:0/3959790724 shutdown_connections 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 --2- 172.21.15.113:0/3959790724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b80cee20 0x7f31b80b5300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 --2- 172.21.15.113:0/3959790724 >> v2:172.21.15.124:3300/0 conn(0x7f31b81551c0 0x7f31b80ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 --2- 172.21.15.113:0/3959790724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b815b390 0x7f31b80b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 -- 172.21.15.113:0/3959790724 >> 172.21.15.113:0/3959790724 conn(0x7f31b8074170 msgr2=0x7f31b806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 -- 172.21.15.113:0/3959790724 shutdown_connections 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.152+0000 7f31bff7a700 1 -- 172.21.15.113:0/3959790724 wait complete. 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.153+0000 7f31bff7a700 1 Processor -- start 2024-01-16T13:52:31.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.153+0000 7f31bff7a700 1 -- start start 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bff7a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b80cee20 0x7f31b813ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bff7a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 0x7f31b8142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bff7a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31b815b390 0x7f31b8159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bff7a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b81f3bc0 con 0x7f31b81551c0 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bff7a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b81456c0 con 0x7f31b815b390 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bff7a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b81438c0 con 0x7f31b80cee20 2024-01-16T13:52:31.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 0x7f31b8142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 0x7f31b8142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41712/0 (socket says 172.21.15.113:41712) 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 -- 172.21.15.113:0/1740541097 learned_addr learned my addr 172.21.15.113:0/1740541097 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31be517700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31b815b390 0x7f31b8159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bdd16700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b80cee20 0x7f31b813ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 -- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b80cee20 msgr2=0x7f31b813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b80cee20 0x7f31b813ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 -- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31b815b390 msgr2=0x7f31b8159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31b815b390 0x7f31b8159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.154+0000 7f31bd515700 1 -- 172.21.15.113:0/1740541097 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0009b30 con 0x7f31b81551c0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.155+0000 7f31bd515700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 0x7f31b8142de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f31a800c7f0 tx=0x7f31a800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.155+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8011800 con 0x7f31b81551c0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.155+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a8011960 con 0x7f31b81551c0 2024-01-16T13:52:31.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.155+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b8144d90 con 0x7f31b81551c0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.155+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8011cd0 con 0x7f31b81551c0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.155+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b8145350 con 0x7f31b81551c0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.156+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b80753f0 con 0x7f31b81551c0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.156+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f31a8010350 con 0x7f31b81551c0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.156+0000 7f31ae7fc700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f319c06b350 0x7f319c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.157+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31a8089560 con 0x7f31b81551c0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.157+0000 7f31bdd16700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f319c06b350 0x7f319c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.157+0000 7f31bdd16700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f319c06b350 0x7f319c06d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f31b4006ba0 tx=0x7f31b4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.161+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31a8054460 con 0x7f31b81551c0 2024-01-16T13:52:31.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.352+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31b8145830 con 0x7f31b81551c0 2024-01-16T13:52:31.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.353+0000 7f31ae7fc700 1 -- 172.21.15.113:0/1740541097 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31a8057a80 con 0x7f31b81551c0 2024-01-16T13:52:31.336 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f319c06b350 msgr2=0x7f319c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f319c06b350 0x7f319c06d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f31b4006ba0 tx=0x7f31b4008040 comp rx=0 tx=0).stop 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 msgr2=0x7f31b8142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 0x7f31b8142de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f31a800c7f0 tx=0x7f31a800cb00 comp rx=0 tx=0).stop 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 shutdown_connections 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f319c06b350 0x7f319c06d810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31b815b390 0x7f31b8159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31b80cee20 0x7f31b813ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 --2- 172.21.15.113:0/1740541097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31b81551c0 0x7f31b8142de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.354+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 >> 172.21.15.113:0/1740541097 conn(0x7f31b8074170 msgr2=0x7f31b815ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.355+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 shutdown_connections 2024-01-16T13:52:31.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.355+0000 7f31bff7a700 1 -- 172.21.15.113:0/1740541097 wait complete. 2024-01-16T13:52:31.350 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:31.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.491+0000 7fc1dbefb700 1 Processor -- start 2024-01-16T13:52:31.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.491+0000 7fc1dbefb700 1 -- start start 2024-01-16T13:52:31.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1dbefb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc1d415e110 0x7fc1d406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1dbefb700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc1d406e420 0x7fc1d406e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1dbefb700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40affe0 con 0x7fc1d40afb30 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1dbefb700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d406d980 con 0x7fc1d406e420 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1dbefb700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d40b0260 con 0x7fc1d415e110 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc1d406e420 0x7fc1d406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc1d406e420 0x7fc1d406e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41716/0 (socket says 172.21.15.113:41716) 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 -- 172.21.15.113:0/3948903883 learned_addr learned my addr 172.21.15.113:0/3948903883 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:31.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1d9c97700 1 --2- 172.21.15.113:0/3948903883 >> v2:172.21.15.124:3300/0 conn(0x7fc1d415e110 0x7fc1d406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 -- 172.21.15.113:0/3948903883 >> v2:172.21.15.124:3300/0 conn(0x7fc1d415e110 msgr2=0x7fc1d406f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 --2- 172.21.15.113:0/3948903883 >> v2:172.21.15.124:3300/0 conn(0x7fc1d415e110 0x7fc1d406f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 -- 172.21.15.113:0/3948903883 >> v1:172.21.15.124:6789/0 conn(0x7fc1d40afb30 legacy=0x7fc1d406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 -- 172.21.15.113:0/3948903883 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40b0540 con 0x7fc1d406e420 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1da498700 1 --2- 172.21.15.113:0/3948903883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d406e420 0x7fc1d406e870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4073030 tx=0x7fc1cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f92f72f55253fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1d8c95700 1 -- 172.21.15.113:0/3948903883 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc00e070 con 0x7fc1d406e420 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1d8c95700 1 -- 172.21.15.113:0/3948903883 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc00b910 con 0x7fc1d406e420 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.492+0000 7fc1d8c95700 1 -- 172.21.15.113:0/3948903883 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc00bcc0 con 0x7fc1d406e420 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 -- 172.21.15.113:0/3948903883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d406e420 msgr2=0x7fc1d406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/3948903883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d406e420 0x7fc1d406e870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4073030 tx=0x7fc1cc00b330 comp rx=0 tx=0).stop 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 -- 172.21.15.113:0/3948903883 shutdown_connections 2024-01-16T13:52:31.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/3948903883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d406e420 0x7fc1d406e870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/3948903883 >> v2:172.21.15.124:3300/0 conn(0x7fc1d415e110 0x7fc1d406f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 -- 172.21.15.113:0/3948903883 >> 172.21.15.113:0/3948903883 conn(0x7fc1d40b3250 msgr2=0x7fc1d40d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:31.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 -- 172.21.15.113:0/3948903883 shutdown_connections 2024-01-16T13:52:31.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.493+0000 7fc1dbefb700 1 -- 172.21.15.113:0/3948903883 wait complete. 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.494+0000 7fc1dbefb700 1 Processor -- start 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.494+0000 7fc1dbefb700 1 -- start start 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1dbefb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1d406e420 0x7fc1d40b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1dbefb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc1d40afb30 0x7fc1d40b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1dbefb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 0x7fc1d4145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1dbefb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d4149a90 con 0x7fc1d415e110 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1dbefb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41497b0 con 0x7fc1d40afb30 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1dbefb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41f76b0 con 0x7fc1d406e420 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 0x7fc1d4145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 0x7fc1d4145d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41726/0 (socket says 172.21.15.113:41726) 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 -- 172.21.15.113:0/1737338992 learned_addr learned my addr 172.21.15.113:0/1737338992 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:31.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1d9496700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc1d40afb30 0x7fc1d40b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1d9c97700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1d406e420 0x7fc1d40b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 -- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1d406e420 msgr2=0x7fc1d40b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1d406e420 0x7fc1d40b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 -- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc1d40afb30 msgr2=0x7fc1d40b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc1d40afb30 0x7fc1d40b0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1da498700 1 -- 172.21.15.113:0/1737338992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc009b30 con 0x7fc1d415e110 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.495+0000 7fc1d9496700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc1d40afb30 0x7fc1d40b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.496+0000 7fc1da498700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 0x7fc1d4145d20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4074b60 tx=0x7fc1cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.496+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc0099b0 con 0x7fc1d415e110 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.496+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc0092e0 con 0x7fc1d415e110 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.496+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d406b880 con 0x7fc1d415e110 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.496+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc01c640 con 0x7fc1d415e110 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.496+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d41f7c70 con 0x7fc1d415e110 2024-01-16T13:52:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.497+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d4069020 con 0x7fc1d415e110 2024-01-16T13:52:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.497+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc1cc01c7a0 con 0x7fc1d415e110 2024-01-16T13:52:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.498+0000 7fc1ca7fc700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc1b806b350 0x7fc1b806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.498+0000 7fc1d9c97700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc1b806b350 0x7fc1b806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.498+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1cc08b510 con 0x7fc1d415e110 2024-01-16T13:52:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.498+0000 7fc1d9c97700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc1b806b350 0x7fc1b806d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009ad0 tx=0x7fc1d0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.502+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1cc056410 con 0x7fc1d415e110 2024-01-16T13:52:31.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.695+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc1d40b4100 con 0x7fc1d415e110 2024-01-16T13:52:31.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.695+0000 7fc1ca7fc700 1 -- 172.21.15.113:0/1737338992 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc1cc059a30 con 0x7fc1d415e110 2024-01-16T13:52:31.678 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:31.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc1b806b350 msgr2=0x7fc1b806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc1b806b350 0x7fc1b806d810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009ad0 tx=0x7fc1d0008040 comp rx=0 tx=0).stop 2024-01-16T13:52:31.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 msgr2=0x7fc1d4145d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 0x7fc1d4145d20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4074b60 tx=0x7fc1cc004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 shutdown_connections 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc1b806b350 0x7fc1b806d810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1d415e110 0x7fc1d4145d20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1d406e420 0x7fc1d40b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 --2- 172.21.15.113:0/1737338992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc1d40afb30 0x7fc1d40b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 >> 172.21.15.113:0/1737338992 conn(0x7fc1d40b3250 msgr2=0x7fc1d40aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 shutdown_connections 2024-01-16T13:52:31.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.697+0000 7fc1dbefb700 1 -- 172.21.15.113:0/1737338992 wait complete. 2024-01-16T13:52:31.692 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:31.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.833+0000 7f24f1186700 1 Processor -- start 2024-01-16T13:52:31.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.833+0000 7f24f1186700 1 -- start start 2024-01-16T13:52:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24f1186700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f24ec15b240 0x7f24ec06e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24f1186700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec06eeb0 con 0x7f24ec06e9a0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24f1186700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec06f190 con 0x7f24ec06c2e0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24f1186700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec06f470 con 0x7f24ec15b240 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ea59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f24ec06e9a0 0x7f24ec15b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56034/0 (socket says 172.21.15.113:56034) 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ea59c700 1 -- 172.21.15.113:0/219561389 learned_addr learned my addr 172.21.15.113:0/219561389 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ead9d700 1 --2- 172.21.15.113:0/219561389 >> v2:172.21.15.124:3300/0 conn(0x7f24ec15b240 0x7f24ec06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24e9d9b700 1 -- 172.21.15.113:0/219561389 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816501501 0 0) 0x7f24ec06eeb0 con 0x7f24ec06e9a0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24e9d9b700 1 -- 172.21.15.113:0/219561389 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d40035b0 con 0x7f24ec06e9a0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24e9d9b700 1 -- 172.21.15.113:0/219561389 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965627602 0 0) 0x7f24d40035b0 con 0x7f24ec06e9a0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ead9d700 1 -- 172.21.15.113:0/219561389 >> v1:172.21.15.124:6789/0 conn(0x7f24ec06c2e0 legacy=0x7f24ec1647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ead9d700 1 -- 172.21.15.113:0/219561389 >> v1:172.21.15.113:6789/0 conn(0x7f24ec06e9a0 legacy=0x7f24ec15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ead9d700 1 -- 172.21.15.113:0/219561389 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec06f750 con 0x7f24ec15b240 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.834+0000 7f24ead9d700 1 --2- 172.21.15.113:0/219561389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec15b240 0x7f24ec06e460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009840 tx=0x7f24dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df6c25e36ca9d3ed server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24e9d9b700 1 -- 172.21.15.113:0/219561389 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0001670 con 0x7f24ec15b240 2024-01-16T13:52:31.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24e9d9b700 1 -- 172.21.15.113:0/219561389 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc004030 con 0x7f24ec15b240 2024-01-16T13:52:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24e9d9b700 1 -- 172.21.15.113:0/219561389 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0043a0 con 0x7f24ec15b240 2024-01-16T13:52:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 -- 172.21.15.113:0/219561389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec15b240 msgr2=0x7f24ec06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 --2- 172.21.15.113:0/219561389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec15b240 0x7f24ec06e460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009840 tx=0x7f24dc009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 -- 172.21.15.113:0/219561389 shutdown_connections 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 --2- 172.21.15.113:0/219561389 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec15b240 0x7f24ec06e460 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 -- 172.21.15.113:0/219561389 >> 172.21.15.113:0/219561389 conn(0x7f24ec0d0c00 msgr2=0x7f24ec0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 -- 172.21.15.113:0/219561389 shutdown_connections 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.835+0000 7f24f1186700 1 -- 172.21.15.113:0/219561389 wait complete. 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 Processor -- start 2024-01-16T13:52:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 -- start start 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec06c2e0 0x7f24ec13d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24ec06e9a0 0x7f24ec1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 0x7f24ec0cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec13dae0 con 0x7f24ec15b240 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec13d7f0 con 0x7f24ec06c2e0 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24f1186700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec151430 con 0x7f24ec06e9a0 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.837+0000 7f24eb59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 0x7f24ec0cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 0x7f24ec0cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41732/0 (socket says 172.21.15.113:41732) 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 -- 172.21.15.113:0/1070433655 learned_addr learned my addr 172.21.15.113:0/1070433655 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:31.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24ea59c700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24ec06e9a0 0x7f24ec1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24ead9d700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec06c2e0 0x7f24ec13d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 -- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24ec06e9a0 msgr2=0x7f24ec1557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24ec06e9a0 0x7f24ec1557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 -- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec06c2e0 msgr2=0x7f24ec13d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec06c2e0 0x7f24ec13d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 -- 172.21.15.113:0/1070433655 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0094a0 con 0x7f24ec15b240 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24ea59c700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24ec06e9a0 0x7f24ec1557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24ead9d700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec06c2e0 0x7f24ec13d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24eb59e700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 0x7f24ec0cf580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f24e400cb50 tx=0x7f24e400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4003c00 con 0x7f24ec15b240 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e4003d60 con 0x7f24ec15b240 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec155d10 con 0x7f24ec15b240 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e400ab70 con 0x7f24ec15b240 2024-01-16T13:52:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.838+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec0af6e0 con 0x7f24ec15b240 2024-01-16T13:52:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.839+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec0afcf0 con 0x7f24ec15b240 2024-01-16T13:52:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.840+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f24e400acd0 con 0x7f24ec15b240 2024-01-16T13:52:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.840+0000 7f24d37fe700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f24d806b350 0x7f24d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.840+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24e4089c00 con 0x7f24ec15b240 2024-01-16T13:52:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.840+0000 7f24ead9d700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f24d806b350 0x7f24d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.841+0000 7f24ead9d700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f24d806b350 0x7f24d806d810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0090b0 tx=0x7f24dc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:31.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:31.845+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24ec0afcf0 con 0x7f24ec15b240 2024-01-16T13:52:32.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.033+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24ec0afcf0 con 0x7f24ec15b240 2024-01-16T13:52:32.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.033+0000 7f24d37fe700 1 -- 172.21.15.113:0/1070433655 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24ec0afcf0 con 0x7f24ec15b240 2024-01-16T13:52:32.017 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f24d806b350 msgr2=0x7f24d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f24d806b350 0x7f24d806d810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0090b0 tx=0x7f24dc008030 comp rx=0 tx=0).stop 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 msgr2=0x7f24ec0cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 0x7f24ec0cf580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f24e400cb50 tx=0x7f24e400cf10 comp rx=0 tx=0).stop 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 shutdown_connections 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f24d806b350 0x7f24d806d810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24ec15b240 0x7f24ec0cf580 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24ec06c2e0 0x7f24ec13d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 --2- 172.21.15.113:0/1070433655 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f24ec06e9a0 0x7f24ec1557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.035+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 >> 172.21.15.113:0/1070433655 conn(0x7f24ec0d0c00 msgr2=0x7f24ec0745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.036+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 shutdown_connections 2024-01-16T13:52:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.036+0000 7f24f1186700 1 -- 172.21.15.113:0/1070433655 wait complete. 2024-01-16T13:52:32.031 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:32.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.169+0000 7f063a277700 1 Processor -- start 2024-01-16T13:52:32.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.169+0000 7f063a277700 1 -- start start 2024-01-16T13:52:32.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f063a277700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f06341473a0 0x7f063415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f063a277700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f063406dc20 0x7f0634073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f063a277700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063415eb00 con 0x7f0634147780 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f063a277700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f063415ede0 con 0x7f063406dc20 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f063a277700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063415f0c0 con 0x7f06341473a0 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f06337fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f0634147780 0x7f063406c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56044/0 (socket says 172.21.15.113:56044) 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f06337fe700 1 -- 172.21.15.113:0/1063530371 learned_addr learned my addr 172.21.15.113:0/1063530371 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0633fff700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f06341473a0 0x7f063415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0638814700 1 --2- 172.21.15.113:0/1063530371 >> v2:172.21.15.124:3300/0 conn(0x7f063406dc20 0x7f0634073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214359667 0 0) 0x7f063415eb00 con 0x7f0634147780 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c0035b0 con 0x7f0634147780 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398865675 0 0) 0x7f061c0035b0 con 0x7f0634147780 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f06341473a0 msgr2=0x7f063415e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f06341473a0 0x7f063415e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 >> v2:172.21.15.124:3300/0 conn(0x7f063406dc20 msgr2=0x7f0634073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 --2- 172.21.15.113:0/1063530371 >> v2:172.21.15.124:3300/0 conn(0x7f063406dc20 0x7f0634073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.170+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063415f3a0 con 0x7f0634147780 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f0624001670 con 0x7f0634147780 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 >> v1:172.21.15.113:6789/0 conn(0x7f0634147780 legacy=0x7f063406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c003a00 0x7f061c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f061c0043f0 0x7f061c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f061c004f40 0x7f061c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063415ede0 con 0x7f061c0043f0 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063415f0c0 con 0x7f061c004f40 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c00bf50 con 0x7f061c003a00 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0633fff700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f06341473a0 0x7f063415e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f061c0043f0 0x7f061c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0638814700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f061c004f40 0x7f061c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0633fff700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c003a00 0x7f061c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 -- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c003a00 msgr2=0x7f061c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c003a00 0x7f061c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 -- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f061c004f40 msgr2=0x7f061c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f061c004f40 0x7f061c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 -- 172.21.15.113:0/1063530371 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c0035b0 con 0x7f061c0043f0 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f06337fe700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f061c0043f0 0x7f061c0048d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f062400c2f0 tx=0x7f061c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c84936695fcb17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062400ce30 con 0x7f061c0043f0 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624005960 con 0x7f061c0043f0 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.171+0000 7f0632ffd700 1 -- 172.21.15.113:0/1063530371 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624005d10 con 0x7f061c0043f0 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f063a277700 1 -- 172.21.15.113:0/1063530371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f061c0043f0 msgr2=0x7f061c0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f063a277700 1 --2- 172.21.15.113:0/1063530371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f061c0043f0 0x7f061c0048d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f062400c2f0 tx=0x7f061c005880 comp rx=0 tx=0).stop 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f0638814700 1 -- 172.21.15.113:0/1063530371 reap_dead start 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f063a277700 1 -- 172.21.15.113:0/1063530371 shutdown_connections 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f063a277700 1 -- 172.21.15.113:0/1063530371 >> 172.21.15.113:0/1063530371 conn(0x7f06340b3130 msgr2=0x7f06340d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f063a277700 1 -- 172.21.15.113:0/1063530371 shutdown_connections 2024-01-16T13:52:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.172+0000 7f063a277700 1 -- 172.21.15.113:0/1063530371 wait complete. 2024-01-16T13:52:32.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.173+0000 7f063a277700 1 Processor -- start 2024-01-16T13:52:32.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.173+0000 7f063a277700 1 -- start start 2024-01-16T13:52:32.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f063a277700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c0043f0 0x7f06340ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f063a277700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 0x7f06340ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f063a277700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06340cf0b0 0x7f06340ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f063a277700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340ce910 con 0x7f063415d730 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f063a277700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340cd610 con 0x7f06340cf0b0 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f063a277700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340cd8f0 con 0x7f061c0043f0 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 0x7f06340ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 0x7f06340ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41754/0 (socket says 172.21.15.113:41754) 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 -- 172.21.15.113:0/2431912658 learned_addr learned my addr 172.21.15.113:0/2431912658 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f0633fff700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c0043f0 0x7f06340ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f0638814700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06340cf0b0 0x7f06340ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 -- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c0043f0 msgr2=0x7f06340ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c0043f0 0x7f06340ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 -- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06340cf0b0 msgr2=0x7f06340ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06340cf0b0 0x7f06340ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f06337fe700 1 -- 172.21.15.113:0/2431912658 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340cdbd0 con 0x7f063415d730 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.174+0000 7f0638814700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06340cf0b0 0x7f06340ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.175+0000 7f06337fe700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 0x7f06340ceb70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0624011000 tx=0x7f0624001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.175+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624004870 con 0x7f063415d730 2024-01-16T13:52:32.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.175+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06240049d0 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.175+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06340cdeb0 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.175+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624004d80 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.175+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06341f5000 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.176+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634067f60 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.176+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f06240070d0 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.176+0000 7f0630ff9700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f062006b350 0x7f062006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.176+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f062408ad90 con 0x7f063415d730 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.177+0000 7f0633fff700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f062006b350 0x7f062006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.177+0000 7f0633fff700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f062006b350 0x7f062006d810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f06341f8810 tx=0x7f0628008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.181+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0624055d50 con 0x7f063415d730 2024-01-16T13:52:32.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.373+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f063406c240 con 0x7f063415d730 2024-01-16T13:52:32.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.374+0000 7f0630ff9700 1 -- 172.21.15.113:0/2431912658 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f0624059370 con 0x7f063415d730 2024-01-16T13:52:32.357 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:32.357 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:32.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f062006b350 msgr2=0x7f062006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f062006b350 0x7f062006d810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f06341f8810 tx=0x7f0628008040 comp rx=0 tx=0).stop 2024-01-16T13:52:32.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 msgr2=0x7f06340ceb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 0x7f06340ceb70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0624011000 tx=0x7f0624001a00 comp rx=0 tx=0).stop 2024-01-16T13:52:32.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 shutdown_connections 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f062006b350 0x7f062006d810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06340cf0b0 0x7f06340ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f061c0043f0 0x7f06340ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 --2- 172.21.15.113:0/2431912658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f063415d730 0x7f06340ceb70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 >> 172.21.15.113:0/2431912658 conn(0x7f06340b3130 msgr2=0x7f0634073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 shutdown_connections 2024-01-16T13:52:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.376+0000 7f063a277700 1 -- 172.21.15.113:0/2431912658 wait complete. 2024-01-16T13:52:32.372 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:32.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:32.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.514+0000 7ffa0ec4d700 1 Processor -- start 2024-01-16T13:52:32.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.514+0000 7ffa0ec4d700 1 -- start start 2024-01-16T13:52:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0ec4d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa080746a0 0x7ffa08168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0ec4d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0815e4a0 con 0x7ffa0815e0c0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0ec4d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0806aff0 con 0x7ffa080afb80 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0ec4d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080b0010 con 0x7ffa080746a0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0c9e9700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ffa0815e0c0 0x7ffa080b3bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56058/0 (socket says 172.21.15.113:56058) 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0c9e9700 1 -- 172.21.15.113:0/1826437760 learned_addr learned my addr 172.21.15.113:0/1826437760 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0d1ea700 1 --2- 172.21.15.113:0/1826437760 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa080746a0 0x7ffa08168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307715838 0 0) 0x7ffa0815e4a0 con 0x7ffa0815e0c0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f80035b0 con 0x7ffa0815e0c0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630122719 0 0) 0x7ffa0806aff0 con 0x7ffa080afb80 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa0815e4a0 con 0x7ffa080afb80 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0d1ea700 1 -- 172.21.15.113:0/1826437760 >> v1:172.21.15.124:6789/0 conn(0x7ffa080afb80 legacy=0x7ffa08073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0d1ea700 1 -- 172.21.15.113:0/1826437760 >> v1:172.21.15.113:6789/0 conn(0x7ffa0815e0c0 legacy=0x7ffa080b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0d1ea700 1 -- 172.21.15.113:0/1826437760 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0806b2d0 con 0x7ffa080746a0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa0d1ea700 1 --2- 172.21.15.113:0/1826437760 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa080746a0 0x7ffa08168fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009df0 tx=0x7ffa0000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fee03d8eec55cfe9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.515+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000e070 con 0x7ffa080746a0 2024-01-16T13:52:32.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0000bae0 con 0x7ffa080746a0 2024-01-16T13:52:32.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa077fe700 1 -- 172.21.15.113:0/1826437760 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000be50 con 0x7ffa080746a0 2024-01-16T13:52:32.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1826437760 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa080746a0 msgr2=0x7ffa08168fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1826437760 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa080746a0 0x7ffa08168fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009df0 tx=0x7ffa0000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:32.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1826437760 shutdown_connections 2024-01-16T13:52:32.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1826437760 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa080746a0 0x7ffa08168fa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1826437760 >> 172.21.15.113:0/1826437760 conn(0x7ffa080d0c00 msgr2=0x7ffa080d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1826437760 shutdown_connections 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.516+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1826437760 wait complete. 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.517+0000 7ffa0ec4d700 1 Processor -- start 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.517+0000 7ffa0ec4d700 1 -- start start 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0ec4d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa080746a0 0x7ffa080cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0ec4d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 0x7ffa080cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0ec4d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa0815e0c0 0x7ffa08146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0ec4d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa081f3c60 con 0x7ffa080afb80 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0ec4d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080ce3f0 con 0x7ffa080746a0 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0ec4d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa080ce6d0 con 0x7ffa0815e0c0 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa07fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 0x7ffa080cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa07fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 0x7ffa080cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41756/0 (socket says 172.21.15.113:41756) 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa07fff700 1 -- 172.21.15.113:0/1819259681 learned_addr learned my addr 172.21.15.113:0/1819259681 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0d1ea700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa0815e0c0 0x7ffa08146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.518+0000 7ffa0c9e9700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa080746a0 0x7ffa080cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa07fff700 1 -- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa0815e0c0 msgr2=0x7ffa08146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa07fff700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa0815e0c0 0x7ffa08146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa07fff700 1 -- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa080746a0 msgr2=0x7ffa080cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa07fff700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa080746a0 0x7ffa080cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa07fff700 1 -- 172.21.15.113:0/1819259681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00009a70 con 0x7ffa080afb80 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa07fff700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 0x7ffa080cdba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400c500 tx=0x7ff9f400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40052d0 con 0x7ffa080afb80 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f4005430 con 0x7ffa080afb80 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40038c0 con 0x7ffa080afb80 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa080ce9b0 con 0x7ffa080afb80 2024-01-16T13:52:32.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.519+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa080cef70 con 0x7ffa080afb80 2024-01-16T13:52:32.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.520+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa081f3dd0 con 0x7ffa080afb80 2024-01-16T13:52:32.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.520+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff9f4003a20 con 0x7ffa080afb80 2024-01-16T13:52:32.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.521+0000 7ffa057fa700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ec06b350 0x7ff9ec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.521+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9f4088f80 con 0x7ffa080afb80 2024-01-16T13:52:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.521+0000 7ffa0c9e9700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ec06b350 0x7ff9ec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.521+0000 7ffa0c9e9700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ec06b350 0x7ff9ec06d810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc007fd0 tx=0x7ff9fc007dc0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.525+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9f40bbe90 con 0x7ffa080afb80 2024-01-16T13:52:32.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.716+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa080b26b0 con 0x7ffa080afb80 2024-01-16T13:52:32.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.717+0000 7ffa057fa700 1 -- 172.21.15.113:0/1819259681 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa081f3dd0 con 0x7ffa080afb80 2024-01-16T13:52:32.700 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:32.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ec06b350 msgr2=0x7ff9ec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ec06b350 0x7ff9ec06d810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc007fd0 tx=0x7ff9fc007dc0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 msgr2=0x7ffa080cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 0x7ffa080cdba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400c500 tx=0x7ff9f400c810 comp rx=0 tx=0).stop 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 shutdown_connections 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff9ec06b350 0x7ff9ec06d810 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa0815e0c0 0x7ffa08146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa080746a0 0x7ffa080cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 --2- 172.21.15.113:0/1819259681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa080afb80 0x7ffa080cdba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 >> 172.21.15.113:0/1819259681 conn(0x7ffa080d0c00 msgr2=0x7ffa080ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 shutdown_connections 2024-01-16T13:52:32.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.719+0000 7ffa0ec4d700 1 -- 172.21.15.113:0/1819259681 wait complete. 2024-01-16T13:52:32.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:32.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.855+0000 7fb7f7fae700 1 Processor -- start 2024-01-16T13:52:32.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.856+0000 7fb7f7fae700 1 -- start start 2024-01-16T13:52:32.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f7fae700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7f0158600 0x7fb7f006f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f7fae700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f7fae700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f006e4e0 con 0x7fb7f00b5380 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f7fae700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f006fac0 con 0x7fb7f0158600 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f7fae700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f006e7c0 con 0x7fb7f00b4ef0 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7f0158600 0x7fb7f006f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb7f0158600 0x7fb7f006f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41760/0 (socket says 172.21.15.113:41760) 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 -- 172.21.15.113:0/675995349 learned_addr learned my addr 172.21.15.113:0/675995349 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5549700 1 --2- 172.21.15.113:0/675995349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 -- 172.21.15.113:0/675995349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 msgr2=0x7fb7f015e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 --2- 172.21.15.113:0/675995349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f015e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 -- 172.21.15.113:0/675995349 >> v1:172.21.15.124:6789/0 conn(0x7fb7f00b5380 legacy=0x7fb7f006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 -- 172.21.15.113:0/675995349 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f006eaa0 con 0x7fb7f0158600 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f5d4a700 1 --2- 172.21.15.113:0/675995349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f006f580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40098f0 tx=0x7fb7e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ba6466df4211b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f4d48700 1 -- 172.21.15.113:0/675995349 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4004230 con 0x7fb7f0158600 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f4d48700 1 -- 172.21.15.113:0/675995349 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e4004390 con 0x7fb7f0158600 2024-01-16T13:52:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.857+0000 7fb7f4d48700 1 -- 172.21.15.113:0/675995349 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40036a0 con 0x7fb7f0158600 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 -- 172.21.15.113:0/675995349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 msgr2=0x7fb7f006f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/675995349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f006f580 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7e40098f0 tx=0x7fb7e4009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 -- 172.21.15.113:0/675995349 shutdown_connections 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/675995349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f006f580 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/675995349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f015e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 -- 172.21.15.113:0/675995349 >> 172.21.15.113:0/675995349 conn(0x7fb7f00b3250 msgr2=0x7fb7f00d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 -- 172.21.15.113:0/675995349 shutdown_connections 2024-01-16T13:52:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.858+0000 7fb7f7fae700 1 -- 172.21.15.113:0/675995349 wait complete. 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.859+0000 7fb7f7fae700 1 Processor -- start 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 -- start start 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f013f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb7f00b5380 0x7fb7f013bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f01527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f013f600 con 0x7fb7f0158600 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f015d2d0 con 0x7fb7f00b5380 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.860+0000 7fb7f7fae700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f015d5b0 con 0x7fb7f00b4ef0 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f01527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f01527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41766/0 (socket says 172.21.15.113:41766) 2024-01-16T13:52:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 -- 172.21.15.113:0/1466819461 learned_addr learned my addr 172.21.15.113:0/1466819461 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f5d4a700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f013f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f5549700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb7f00b5380 0x7fb7f013bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 -- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 msgr2=0x7fb7f013f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f013f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 -- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb7f00b5380 msgr2=0x7fb7f013bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb7f00b5380 0x7fb7f013bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 -- 172.21.15.113:0/1466819461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4009550 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f654b700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f01527e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800cba0 tx=0x7fb7e800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e800ab80 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e800ace0 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f015d890 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8012660 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.861+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f015de50 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.862+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f0069020 con 0x7fb7f0158600 2024-01-16T13:52:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.863+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb7e80127c0 con 0x7fb7f0158600 2024-01-16T13:52:32.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.863+0000 7fb7e27fc700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb7d406b350 0x7fb7d406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:32.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.863+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7e808a3c0 con 0x7fb7f0158600 2024-01-16T13:52:32.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.863+0000 7fb7f5d4a700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb7d406b350 0x7fb7d406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:32.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.863+0000 7fb7f5d4a700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb7d406b350 0x7fb7d406d810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009130 tx=0x7fb7e4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:32.867+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7e8055230 con 0x7fb7f0158600 2024-01-16T13:52:33.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.059+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7f006e190 con 0x7fb7f0158600 2024-01-16T13:52:33.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.060+0000 7fb7e27fc700 1 -- 172.21.15.113:0/1466819461 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7e8058850 con 0x7fb7f0158600 2024-01-16T13:52:33.043 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:33.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb7d406b350 msgr2=0x7fb7d406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb7d406b350 0x7fb7d406d810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009130 tx=0x7fb7e4008030 comp rx=0 tx=0).stop 2024-01-16T13:52:33.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 msgr2=0x7fb7f01527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f01527e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800cba0 tx=0x7fb7e800cf60 comp rx=0 tx=0).stop 2024-01-16T13:52:33.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 shutdown_connections 2024-01-16T13:52:33.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb7d406b350 0x7fb7d406d810 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0158600 0x7fb7f01527e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb7f00b4ef0 0x7fb7f013f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 --2- 172.21.15.113:0/1466819461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb7f00b5380 0x7fb7f013bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 >> 172.21.15.113:0/1466819461 conn(0x7fb7f00b3250 msgr2=0x7fb7f00b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 shutdown_connections 2024-01-16T13:52:33.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.062+0000 7fb7f7fae700 1 -- 172.21.15.113:0/1466819461 wait complete. 2024-01-16T13:52:33.057 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:33.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.198+0000 7fe156a05700 1 Processor -- start 2024-01-16T13:52:33.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.198+0000 7fe156a05700 1 -- start start 2024-01-16T13:52:33.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe156a05700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe1500cffc0 0x7fe1500b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe156a05700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b3860 con 0x7fe15015f8d0 2024-01-16T13:52:33.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe156a05700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500b2520 con 0x7fe1500d03a0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe156a05700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1500b2800 con 0x7fe1500cffc0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe154fa2700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe15015f8d0 0x7fe1500ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56074/0 (socket says 172.21.15.113:56074) 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe154fa2700 1 -- 172.21.15.113:0/4097892244 learned_addr learned my addr 172.21.15.113:0/4097892244 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787164813 0 0) 0x7fe148001490 con 0x7fe15015f8d0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14ffff700 1 --2- 172.21.15.113:0/4097892244 >> v2:172.21.15.124:3300/0 conn(0x7fe1500cffc0 0x7fe1500b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1380035b0 con 0x7fe15015f8d0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4290598002 0 0) 0x7fe1380035b0 con 0x7fe15015f8d0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 >> v2:172.21.15.124:3300/0 conn(0x7fe1500cffc0 msgr2=0x7fe1500b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 --2- 172.21.15.113:0/4097892244 >> v2:172.21.15.124:3300/0 conn(0x7fe1500cffc0 0x7fe1500b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 >> v1:172.21.15.124:6789/0 conn(0x7fe1500d03a0 legacy=0x7fe15015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1500b2ae0 con 0x7fe15015f8d0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fe148001b80 con 0x7fe15015f8d0 2024-01-16T13:52:33.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 >> v1:172.21.15.113:6789/0 conn(0x7fe15015f8d0 legacy=0x7fe1500ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe138003a00 0x7fe138003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138004850 0x7fe138004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe138005210 0x7fe13800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500b2800 con 0x7fe138003a00 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13800c1b0 con 0x7fe138004850 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.199+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13800c370 con 0x7fe138005210 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe138003a00 0x7fe138003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe154fa2700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe138005210 0x7fe13800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14f7fe700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138004850 0x7fe138004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 -- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe138005210 msgr2=0x7fe13800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe138005210 0x7fe13800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 -- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138004850 msgr2=0x7fe138004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138004850 0x7fe138004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 -- 172.21.15.113:0/4097892244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1380035b0 con 0x7fe138003a00 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe154fa2700 1 -- 172.21.15.113:0/4097892244 reap_dead start 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14ffff700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe138003a00 0x7fe138003e20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe144009b60 tx=0x7fe138005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52d061269230679 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14400e6c0 con 0x7fe138003a00 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14400e820 con 0x7fe138003a00 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.200+0000 7fe14effd700 1 -- 172.21.15.113:0/4097892244 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14400eb90 con 0x7fe138003a00 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 -- 172.21.15.113:0/4097892244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe138003a00 msgr2=0x7fe138003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe138003a00 0x7fe138003e20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe144009b60 tx=0x7fe138005880 comp rx=0 tx=0).stop 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 -- 172.21.15.113:0/4097892244 shutdown_connections 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 --2- 172.21.15.113:0/4097892244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe138003a00 0x7fe138003e20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 -- 172.21.15.113:0/4097892244 >> 172.21.15.113:0/4097892244 conn(0x7fe1500d1e50 msgr2=0x7fe1500d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 -- 172.21.15.113:0/4097892244 shutdown_connections 2024-01-16T13:52:33.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.201+0000 7fe156a05700 1 -- 172.21.15.113:0/4097892244 wait complete. 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.202+0000 7fe156a05700 1 Processor -- start 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.202+0000 7fe156a05700 1 -- start start 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe156a05700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138003a00 0x7fe1500b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe156a05700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1500b6730 0x7fe1500b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe156a05700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 0x7fe1501f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe156a05700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15006c800 con 0x7fe1500b70d0 2024-01-16T13:52:33.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe156a05700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500b7550 con 0x7fe138003a00 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe156a05700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15006bb80 con 0x7fe1500b6730 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe154fa2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 0x7fe1501f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe154fa2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 0x7fe1501f4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41782/0 (socket says 172.21.15.113:41782) 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.203+0000 7fe154fa2700 1 -- 172.21.15.113:0/2177694205 learned_addr learned my addr 172.21.15.113:0/2177694205 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe14f7fe700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1500b6730 0x7fe1500b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe14ffff700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138003a00 0x7fe1500b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe154fa2700 1 -- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1500b6730 msgr2=0x7fe1500b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe154fa2700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1500b6730 0x7fe1500b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe154fa2700 1 -- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138003a00 msgr2=0x7fe1500b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe154fa2700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138003a00 0x7fe1500b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe154fa2700 1 -- 172.21.15.113:0/2177694205 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15006be60 con 0x7fe1500b70d0 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe154fa2700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 0x7fe1501f4f40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe148000f80 tx=0x7fe148007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1480079a0 con 0x7fe1500b70d0 2024-01-16T13:52:33.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe148007b20 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15006c140 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148013630 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.204+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe150070680 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.205+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15006c940 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.205+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe148013790 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.206+0000 7fe14cff9700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe13c06b350 0x7fe13c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.206+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe14808a1f0 con 0x7fe1500b70d0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.206+0000 7fe14ffff700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe13c06b350 0x7fe13c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.206+0000 7fe14ffff700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe13c06b350 0x7fe13c06d810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe1440090b0 tx=0x7fe14401e040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.210+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe15006c940 con 0x7fe1500b70d0 2024-01-16T13:52:33.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.398+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe15006c940 con 0x7fe1500b70d0 2024-01-16T13:52:33.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.399+0000 7fe14cff9700 1 -- 172.21.15.113:0/2177694205 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe15006c940 con 0x7fe1500b70d0 2024-01-16T13:52:33.382 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe13c06b350 msgr2=0x7fe13c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe13c06b350 0x7fe13c06d810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe1440090b0 tx=0x7fe14401e040 comp rx=0 tx=0).stop 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 msgr2=0x7fe1501f4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 0x7fe1501f4f40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe148000f80 tx=0x7fe148007340 comp rx=0 tx=0).stop 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 shutdown_connections 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe13c06b350 0x7fe13c06d810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1500b70d0 0x7fe1501f4f40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe138003a00 0x7fe1500b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 --2- 172.21.15.113:0/2177694205 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1500b6730 0x7fe1500b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 >> 172.21.15.113:0/2177694205 conn(0x7fe1500d1e50 msgr2=0x7fe15006efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:33.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 shutdown_connections 2024-01-16T13:52:33.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.401+0000 7fe156a05700 1 -- 172.21.15.113:0/2177694205 wait complete. 2024-01-16T13:52:33.475 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-16T13:52:33.475 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T13:52:33.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:33.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.615+0000 7f25a088a700 1 Processor -- start 2024-01-16T13:52:33.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.615+0000 7f25a088a700 1 -- start start 2024-01-16T13:52:33.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f25a088a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f259c15e080 0x7f259c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f25a088a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f259c0ce4d0 0x7f259c0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f25a088a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c0d0150 con 0x7f259c0ce910 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f25a088a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c15e460 con 0x7f259c0ce4d0 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f25a088a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259c0cee50 con 0x7f259c15e080 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f259c0ce4d0 0x7f259c0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f259c0ce4d0 0x7f259c0d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55864/0 (socket says 172.21.15.113:55864) 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 -- 172.21.15.113:0/1565685694 learned_addr learned my addr 172.21.15.113:0/1565685694 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f259a59c700 1 --2- 172.21.15.113:0/1565685694 >> v2:172.21.15.124:3300/0 conn(0x7f259c15e080 0x7f259c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 -- 172.21.15.113:0/1565685694 >> v2:172.21.15.124:3300/0 conn(0x7f259c15e080 msgr2=0x7f259c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 --2- 172.21.15.113:0/1565685694 >> v2:172.21.15.124:3300/0 conn(0x7f259c15e080 0x7f259c0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 -- 172.21.15.113:0/1565685694 >> v1:172.21.15.124:6789/0 conn(0x7f259c0ce910 legacy=0x7f259c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f2599d9b700 1 -- 172.21.15.113:0/1565685694 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0cf130 con 0x7f259c0ce4d0 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.616+0000 7f259a59c700 1 --2- 172.21.15.113:0/1565685694 >> v2:172.21.15.124:3300/0 conn(0x7f259c15e080 0x7f259c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f2599d9b700 1 --2- 172.21.15.113:0/1565685694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0d1be0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f259c073070 tx=0x7f2590009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46da8470ccf1294b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f259959a700 1 -- 172.21.15.113:0/1565685694 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259000b870 con 0x7f259c0ce4d0 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f259959a700 1 -- 172.21.15.113:0/1565685694 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259000b9d0 con 0x7f259c0ce4d0 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f259959a700 1 -- 172.21.15.113:0/1565685694 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259000bd60 con 0x7f259c0ce4d0 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 -- 172.21.15.113:0/1565685694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 msgr2=0x7f259c0d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 --2- 172.21.15.113:0/1565685694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0d1be0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f259c073070 tx=0x7f2590009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 -- 172.21.15.113:0/1565685694 shutdown_connections 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 --2- 172.21.15.113:0/1565685694 >> v2:172.21.15.124:3300/0 conn(0x7f259c15e080 0x7f259c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 --2- 172.21.15.113:0/1565685694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0d1be0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 -- 172.21.15.113:0/1565685694 >> 172.21.15.113:0/1565685694 conn(0x7f259c073130 msgr2=0x7f259c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 -- 172.21.15.113:0/1565685694 shutdown_connections 2024-01-16T13:52:33.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.617+0000 7f25a088a700 1 -- 172.21.15.113:0/1565685694 wait complete. 2024-01-16T13:52:33.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 Processor -- start 2024-01-16T13:52:33.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 -- start start 2024-01-16T13:52:33.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f259c0ce910 0x7f259c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f259c15e080 0x7f259c0b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0ca350 con 0x7f259c0ce4d0 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0c96d0 con 0x7f259c0ce910 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.619+0000 7f25a088a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c0c99b0 con 0x7f259c15e080 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55880/0 (socket says 172.21.15.113:55880) 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 -- 172.21.15.113:0/2108371422 learned_addr learned my addr 172.21.15.113:0/2108371422 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f2599d9b700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f259c0ce910 0x7f259c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259ad9d700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f259c15e080 0x7f259c0b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 -- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f259c15e080 msgr2=0x7f259c0b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f259c15e080 0x7f259c0b6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 -- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f259c0ce910 msgr2=0x7f259c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f259c0ce910 0x7f259c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 -- 172.21.15.113:0/2108371422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2590009550 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f259a59c700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0c5760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e810 tx=0x7f258c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c013610 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c013770 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c013b20 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c0c9c90 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.620+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c09f3b0 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.621+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c0ca5c0 con 0x7f259c0ce4d0 2024-01-16T13:52:33.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.622+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f258c013c80 con 0x7f259c0ce4d0 2024-01-16T13:52:33.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.622+0000 7f258affd700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f257c06b350 0x7f257c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.622+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f258c089800 con 0x7f259c0ce4d0 2024-01-16T13:52:33.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.622+0000 7f2599d9b700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f257c06b350 0x7f257c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.622+0000 7f2599d9b700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f257c06b350 0x7f257c06d810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2590014040 tx=0x7f2590008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.627+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f258c054700 con 0x7f259c0ce4d0 2024-01-16T13:52:33.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.816+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f259c067f60 con 0x7f259c0ce4d0 2024-01-16T13:52:33.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.817+0000 7f258affd700 1 -- 172.21.15.113:0/2108371422 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f258c021090 con 0x7f259c0ce4d0 2024-01-16T13:52:33.801 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:33.801 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:33.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f257c06b350 msgr2=0x7f257c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f257c06b350 0x7f257c06d810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2590014040 tx=0x7f2590008030 comp rx=0 tx=0).stop 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 msgr2=0x7f259c0c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0c5760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f258c00e810 tx=0x7f258c00ebd0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 shutdown_connections 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f257c06b350 0x7f257c06d810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f259c15e080 0x7f259c0b6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f259c0ce4d0 0x7f259c0c5760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 --2- 172.21.15.113:0/2108371422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f259c0ce910 0x7f259c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 >> 172.21.15.113:0/2108371422 conn(0x7f259c073130 msgr2=0x7f259c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 shutdown_connections 2024-01-16T13:52:33.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.819+0000 7f25a088a700 1 -- 172.21.15.113:0/2108371422 wait complete. 2024-01-16T13:52:33.814 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:33.815 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:33.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.956+0000 7f09d92da700 1 Processor -- start 2024-01-16T13:52:33.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.956+0000 7f09d92da700 1 -- start start 2024-01-16T13:52:33.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.956+0000 7f09d92da700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 0x7f09d40b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.956+0000 7f09d92da700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4161d10 con 0x7f09d415bb30 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.956+0000 7f09d92da700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4161ff0 con 0x7f09d40b1740 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.956+0000 7f09d92da700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4162990 con 0x7f09d406bbb0 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d2ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f09d415bb30 0x7f09d415f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38674/0 (socket says 172.21.15.113:38674) 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d2ffd700 1 -- 172.21.15.113:0/4067476647 learned_addr learned my addr 172.21.15.113:0/4067476647 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d37fe700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 0x7f09d40b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454952625 0 0) 0x7f09d4161d10 con 0x7f09d415bb30 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09bc0035b0 con 0x7f09d415bb30 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2788261848 0 0) 0x7f09bc0035b0 con 0x7f09d415bb30 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 msgr2=0x7f09d40b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 0x7f09d40b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 >> v1:172.21.15.124:6789/0 conn(0x7f09d40b1740 legacy=0x7f09d406b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d41622d0 con 0x7f09d415bb30 2024-01-16T13:52:33.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f09c4001670 con 0x7f09d415bb30 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 >> v1:172.21.15.113:6789/0 conn(0x7f09d415bb30 legacy=0x7f09d415f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09bc003a30 0x7f09bc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 0x7f09bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc004d90 0x7f09bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4162990 con 0x7f09bc003a30 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc00bc00 con 0x7f09bc004d90 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc00bdc0 con 0x7f09d406bbb0 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09bc003a30 0x7f09bc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d27fc700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 0x7f09bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.957+0000 7f09d37fe700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc004d90 0x7f09bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d2ffd700 1 -- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 msgr2=0x7f09bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d406bbb0 0x7f09bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d2ffd700 1 -- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc004d90 msgr2=0x7f09bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:33.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc004d90 0x7f09bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d2ffd700 1 -- 172.21.15.113:0/4067476647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0035b0 con 0x7f09bc003a30 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09bc003a30 0x7f09bc003e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f09c400c240 tx=0x7f09bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2197d33266f64835 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4007c10 con 0x7f09bc003a30 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c400cde0 con 0x7f09bc003a30 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d1ffb700 1 -- 172.21.15.113:0/4067476647 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4005ba0 con 0x7f09bc003a30 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d92da700 1 -- 172.21.15.113:0/4067476647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09bc003a30 msgr2=0x7f09bc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d92da700 1 --2- 172.21.15.113:0/4067476647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09bc003a30 0x7f09bc003e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f09c400c240 tx=0x7f09bc005880 comp rx=0 tx=0).stop 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d37fe700 1 -- 172.21.15.113:0/4067476647 reap_dead start 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d92da700 1 -- 172.21.15.113:0/4067476647 shutdown_connections 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d92da700 1 -- 172.21.15.113:0/4067476647 >> 172.21.15.113:0/4067476647 conn(0x7f09d409aa00 msgr2=0x7f09d409ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d92da700 1 -- 172.21.15.113:0/4067476647 shutdown_connections 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.958+0000 7f09d92da700 1 -- 172.21.15.113:0/4067476647 wait complete. 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.960+0000 7f09d92da700 1 Processor -- start 2024-01-16T13:52:33.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.960+0000 7f09d92da700 1 -- start start 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc003a30 0x7f09d415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 0x7f09d40cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d40c64a0 0x7f09d40c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40d0e10 con 0x7f09d415b1e0 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415b060 con 0x7f09bc003a30 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40c6e60 con 0x7f09d40c64a0 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 0x7f09d40cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 0x7f09d40cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55898/0 (socket says 172.21.15.113:55898) 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 -- 172.21.15.113:0/761899592 learned_addr learned my addr 172.21.15.113:0/761899592 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc003a30 0x7f09d415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d37fe700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d40c64a0 0x7f09d40c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 -- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d40c64a0 msgr2=0x7f09d40c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d40c64a0 0x7f09d40c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 -- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc003a30 msgr2=0x7f09d415ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc003a30 0x7f09d415ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 -- 172.21.15.113:0/761899592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40cba50 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d27fc700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 0x7f09d40cb200 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f09c800f820 tx=0x7f09c800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80052e0 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c8005440 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d40cbd30 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c80038c0 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.961+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d40d0a30 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.962+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d40d0bf0 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.963+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09c8003a20 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.963+0000 7f09bb7fe700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f09c006b350 0x7f09c006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.963+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09c80897d0 con 0x7f09d415b1e0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.963+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f09c006b350 0x7f09c006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:33.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.963+0000 7f09d2ffd700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f09c006b350 0x7f09c006d810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09c4000c00 tx=0x7f09c4007000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:33.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:33.967+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09c8054640 con 0x7f09d415b1e0 2024-01-16T13:52:34.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.156+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09d40d0f50 con 0x7f09d415b1e0 2024-01-16T13:52:34.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.157+0000 7f09bb7fe700 1 -- 172.21.15.113:0/761899592 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09d40d0f50 con 0x7f09d415b1e0 2024-01-16T13:52:34.140 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f09c006b350 msgr2=0x7f09c006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f09c006b350 0x7f09c006d810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09c4000c00 tx=0x7f09c4007000 comp rx=0 tx=0).stop 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 msgr2=0x7f09d40cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 0x7f09d40cb200 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f09c800f820 tx=0x7f09c800fb30 comp rx=0 tx=0).stop 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 shutdown_connections 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f09c006b350 0x7f09c006d810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09d40c64a0 0x7f09d40c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09bc003a30 0x7f09d415ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 --2- 172.21.15.113:0/761899592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d415b1e0 0x7f09d40cb200 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 >> 172.21.15.113:0/761899592 conn(0x7f09d409aa00 msgr2=0x7f09d415da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 shutdown_connections 2024-01-16T13:52:34.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.159+0000 7f09d92da700 1 -- 172.21.15.113:0/761899592 wait complete. 2024-01-16T13:52:34.154 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:34.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.295+0000 7f2c47617700 1 Processor -- start 2024-01-16T13:52:34.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.295+0000 7f2c47617700 1 -- start start 2024-01-16T13:52:34.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c47617700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c400cf740 0x7f2c4006db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c47617700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2c401603c0 0x7f2c4006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c47617700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4006f5e0 con 0x7f2c4006c540 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c47617700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4006fba0 con 0x7f2c401603c0 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c47617700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4006c9d0 con 0x7f2c400cf740 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c44bb2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2c401603c0 0x7f2c4006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c453b3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c400cf740 0x7f2c4006db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c44bb2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2c401603c0 0x7f2c4006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:46726/0 (socket says 172.21.15.113:46726) 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c44bb2700 1 -- 172.21.15.113:0/3159800967 learned_addr learned my addr 172.21.15.113:0/3159800967 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c45bb4700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f2c4006c540 0x7f2c40155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:37842/0 (socket says 172.21.15.113:37842) 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c453b3700 1 -- 172.21.15.113:0/3159800967 >> v2:172.21.15.124:3300/0 conn(0x7f2c401603c0 msgr2=0x7f2c4006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c453b3700 1 --2- 172.21.15.113:0/3159800967 >> v2:172.21.15.124:3300/0 conn(0x7f2c401603c0 0x7f2c4006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c37fff700 1 -- 172.21.15.113:0/3159800967 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780944446 0 0) 0x7f2c4006f5e0 con 0x7f2c4006c540 2024-01-16T13:52:34.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c453b3700 1 -- 172.21.15.113:0/3159800967 >> v1:172.21.15.124:6789/0 conn(0x7f2c4006c540 legacy=0x7f2c40155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.296+0000 7f2c453b3700 1 -- 172.21.15.113:0/3159800967 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4006f8c0 con 0x7f2c400cf740 2024-01-16T13:52:34.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c453b3700 1 --2- 172.21.15.113:0/3159800967 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c400cf740 0x7f2c4006db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009840 tx=0x7f2c30009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5281d1a1c7cdda0f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c37fff700 1 -- 172.21.15.113:0/3159800967 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30011830 con 0x7f2c400cf740 2024-01-16T13:52:34.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c37fff700 1 -- 172.21.15.113:0/3159800967 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c30011990 con 0x7f2c400cf740 2024-01-16T13:52:34.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c37fff700 1 -- 172.21.15.113:0/3159800967 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30011d20 con 0x7f2c400cf740 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 -- 172.21.15.113:0/3159800967 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c400cf740 msgr2=0x7f2c4006db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 --2- 172.21.15.113:0/3159800967 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c400cf740 0x7f2c4006db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009840 tx=0x7f2c30009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 -- 172.21.15.113:0/3159800967 shutdown_connections 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 --2- 172.21.15.113:0/3159800967 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c400cf740 0x7f2c4006db80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 --2- 172.21.15.113:0/3159800967 >> v2:172.21.15.124:3300/0 conn(0x7f2c401603c0 0x7f2c4006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 -- 172.21.15.113:0/3159800967 >> 172.21.15.113:0/3159800967 conn(0x7f2c400ce500 msgr2=0x7f2c40071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 -- 172.21.15.113:0/3159800967 shutdown_connections 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.297+0000 7f2c47617700 1 -- 172.21.15.113:0/3159800967 wait complete. 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 Processor -- start 2024-01-16T13:52:34.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 -- start start 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c4006c540 0x7f2c401e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2c400cf740 0x7f2c400af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 0x7f2c401480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401f1fe0 con 0x7f2c401603c0 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401f22c0 con 0x7f2c400cf740 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c47617700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401f25a0 con 0x7f2c4006c540 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.299+0000 7f2c45bb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 0x7f2c401480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 0x7f2c401480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55912/0 (socket says 172.21.15.113:55912) 2024-01-16T13:52:34.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 -- 172.21.15.113:0/924607706 learned_addr learned my addr 172.21.15.113:0/924607706 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c453b3700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c4006c540 0x7f2c401e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c44bb2700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2c400cf740 0x7f2c400af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 -- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c4006c540 msgr2=0x7f2c401e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c4006c540 0x7f2c401e6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 -- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2c400cf740 msgr2=0x7f2c400af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2c400cf740 0x7f2c400af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 -- 172.21.15.113:0/924607706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300094a0 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c45bb4700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 0x7f2c401480d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800cb60 tx=0x7f2c3800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3800a9e0 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3800ab40 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c401f2880 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c380070d0 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.300+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c401e66b0 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.301+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c400cf2a0 con 0x7f2c401603c0 2024-01-16T13:52:34.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.301+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2c38007230 con 0x7f2c401603c0 2024-01-16T13:52:34.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.302+0000 7f2c35ffb700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2c2406b350 0x7f2c2406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.302+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c38089870 con 0x7f2c401603c0 2024-01-16T13:52:34.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.302+0000 7f2c453b3700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2c2406b350 0x7f2c2406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.302+0000 7f2c453b3700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2c2406b350 0x7f2c2406d810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2c30000c00 tx=0x7f2c3001c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.306+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c380546e0 con 0x7f2c401603c0 2024-01-16T13:52:34.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.498+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c40069020 con 0x7f2c401603c0 2024-01-16T13:52:34.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.498+0000 7f2c35ffb700 1 -- 172.21.15.113:0/924607706 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c380180a0 con 0x7f2c401603c0 2024-01-16T13:52:34.482 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2c2406b350 msgr2=0x7f2c2406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2c2406b350 0x7f2c2406d810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2c30000c00 tx=0x7f2c3001c040 comp rx=0 tx=0).stop 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 msgr2=0x7f2c401480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 0x7f2c401480d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800cb60 tx=0x7f2c3800ce70 comp rx=0 tx=0).stop 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 shutdown_connections 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2c2406b350 0x7f2c2406d810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c401603c0 0x7f2c401480d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2c4006c540 0x7f2c401e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 --2- 172.21.15.113:0/924607706 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2c400cf740 0x7f2c400af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 >> 172.21.15.113:0/924607706 conn(0x7f2c400ce500 msgr2=0x7f2c400b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:34.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 shutdown_connections 2024-01-16T13:52:34.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.501+0000 7f2c47617700 1 -- 172.21.15.113:0/924607706 wait complete. 2024-01-16T13:52:34.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:34.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.639+0000 7fbdd0a41700 1 Processor -- start 2024-01-16T13:52:34.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.639+0000 7fbdd0a41700 1 -- start start 2024-01-16T13:52:34.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.640+0000 7fbdd0a41700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbdcc15b240 0x7fbdcc06e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.640+0000 7fbdd0a41700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdcc06eeb0 con 0x7fbdcc06e9a0 2024-01-16T13:52:34.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.640+0000 7fbdd0a41700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdcc06f190 con 0x7fbdcc06c2e0 2024-01-16T13:52:34.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.640+0000 7fbdd0a41700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdcc06f470 con 0x7fbdcc15b240 2024-01-16T13:52:34.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc9d9b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fbdcc06e9a0 0x7fbdcc15b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38690/0 (socket says 172.21.15.113:38690) 2024-01-16T13:52:34.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc9d9b700 1 -- 172.21.15.113:0/761994283 learned_addr learned my addr 172.21.15.113:0/761994283 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdca59c700 1 --2- 172.21.15.113:0/761994283 >> v2:172.21.15.124:3300/0 conn(0x7fbdcc15b240 0x7fbdcc06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245625820 0 0) 0x7fbdcc06eeb0 con 0x7fbdcc06e9a0 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb40035b0 con 0x7fbdcc06e9a0 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2653836535 0 0) 0x7fbdb40035b0 con 0x7fbdcc06e9a0 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 >> v2:172.21.15.124:3300/0 conn(0x7fbdcc15b240 msgr2=0x7fbdcc06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 --2- 172.21.15.113:0/761994283 >> v2:172.21.15.124:3300/0 conn(0x7fbdcc15b240 0x7fbdcc06e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 >> v1:172.21.15.124:6789/0 conn(0x7fbdcc06c2e0 legacy=0x7fbdcc1647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc06f750 con 0x7fbdcc06e9a0 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fbdbc001670 con 0x7fbdcc06e9a0 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 >> v1:172.21.15.113:6789/0 conn(0x7fbdcc06e9a0 legacy=0x7fbdcc15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdb4003a00 0x7fbdb4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4004850 0x7fbdb4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdb4005210 0x7fbdb400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc06f470 con 0x7fbdb4005210 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb400c0b0 con 0x7fbdb4004850 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb400c270 con 0x7fbdb4003a00 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdcad9d700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdb4005210 0x7fbdb400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdc9d9b700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4004850 0x7fbdb4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdca59c700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdb4003a00 0x7fbdb4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdcad9d700 1 -- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdb4003a00 msgr2=0x7fbdb4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdcad9d700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdb4003a00 0x7fbdb4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdcad9d700 1 -- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4004850 msgr2=0x7fbdb4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdcad9d700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4004850 0x7fbdb4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.641+0000 7fbdcad9d700 1 -- 172.21.15.113:0/761994283 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb40035b0 con 0x7fbdb4005210 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdcad9d700 1 -- 172.21.15.113:0/761994283 reap_dead start 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdcad9d700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdb4005210 0x7fbdb400ba10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c190 tx=0x7fbdb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ab5f04232c3bbf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4001db0 con 0x7fbdb4005210 2024-01-16T13:52:34.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc400ccf0 con 0x7fbdb4005210 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdc959a700 1 -- 172.21.15.113:0/761994283 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4003c40 con 0x7fbdb4005210 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 -- 172.21.15.113:0/761994283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdb4005210 msgr2=0x7fbdb400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdb4005210 0x7fbdb400ba10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c190 tx=0x7fbdb4005880 comp rx=0 tx=0).stop 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 -- 172.21.15.113:0/761994283 shutdown_connections 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/761994283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdb4005210 0x7fbdb400ba10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 -- 172.21.15.113:0/761994283 >> 172.21.15.113:0/761994283 conn(0x7fbdcc0d0c00 msgr2=0x7fbdcc0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 -- 172.21.15.113:0/761994283 shutdown_connections 2024-01-16T13:52:34.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.642+0000 7fbdd0a41700 1 -- 172.21.15.113:0/761994283 wait complete. 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 Processor -- start 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 -- start start 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4005210 0x7fbdcc13f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 0x7fbdcc153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdcc0cf1f0 0x7fbdcc0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc154370 con 0x7fbdcc1539b0 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc13fad0 con 0x7fbdb4005210 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.644+0000 7fbdd0a41700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc0cfbb0 con 0x7fbdcc0cf1f0 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 0x7fbdcc153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 0x7fbdcc153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55932/0 (socket says 172.21.15.113:55932) 2024-01-16T13:52:34.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 -- 172.21.15.113:0/393754865 learned_addr learned my addr 172.21.15.113:0/393754865 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdca59c700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4005210 0x7fbdcc13f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdcad9d700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdcc0cf1f0 0x7fbdcc0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 -- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdcc0cf1f0 msgr2=0x7fbdcc0cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdcc0cf1f0 0x7fbdcc0cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 -- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4005210 msgr2=0x7fbdcc13f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4005210 0x7fbdcc13f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 -- 172.21.15.113:0/393754865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc0cfe60 con 0x7fbdcc1539b0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdc9d9b700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 0x7fbdcc153e30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0072c0 tx=0x7fbdbc0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc0077b0 con 0x7fbdcc1539b0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc00f8d0 con 0x7fbdcc1539b0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc00fc80 con 0x7fbdcc1539b0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdcc0d0140 con 0x7fbdcc1539b0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.645+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc14e700 con 0x7fbdcc1539b0 2024-01-16T13:52:34.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.646+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc069020 con 0x7fbdcc1539b0 2024-01-16T13:52:34.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.647+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbdbc01d360 con 0x7fbdcc1539b0 2024-01-16T13:52:34.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.647+0000 7fbdbaffd700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbdac06b380 0x7fbdac06d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.647+0000 7fbdca59c700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbdac06b380 0x7fbdac06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.647+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbdbc089260 con 0x7fbdcc1539b0 2024-01-16T13:52:34.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.648+0000 7fbdca59c700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbdac06b380 0x7fbdac06d840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00045f0 tx=0x7fbdc0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.652+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdbc0540d0 con 0x7fbdcc1539b0 2024-01-16T13:52:34.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.841+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbdcc1499a0 con 0x7fbdcc1539b0 2024-01-16T13:52:34.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.842+0000 7fbdbaffd700 1 -- 172.21.15.113:0/393754865 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbdbc0576f0 con 0x7fbdcc1539b0 2024-01-16T13:52:34.825 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.843+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbdac06b380 msgr2=0x7fbdac06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.843+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbdac06b380 0x7fbdac06d840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00045f0 tx=0x7fbdc0008040 comp rx=0 tx=0).stop 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.843+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 msgr2=0x7fbdcc153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.843+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 0x7fbdcc153e30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0072c0 tx=0x7fbdbc0075d0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 shutdown_connections 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbdac06b380 0x7fbdac06d840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdcc0cf1f0 0x7fbdcc0cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdb4005210 0x7fbdcc13f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 --2- 172.21.15.113:0/393754865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbdcc1539b0 0x7fbdcc153e30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 >> 172.21.15.113:0/393754865 conn(0x7fbdcc0d0c00 msgr2=0x7fbdcc074830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:34.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 shutdown_connections 2024-01-16T13:52:34.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.844+0000 7fbdd0a41700 1 -- 172.21.15.113:0/393754865 wait complete. 2024-01-16T13:52:34.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T13:52:34.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.977+0000 7f887dff2700 1 Processor -- start 2024-01-16T13:52:34.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.977+0000 7f887dff2700 1 -- start start 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.977+0000 7f887dff2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f887815e080 0x7f88780cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.977+0000 7f887dff2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780d0150 con 0x7f88780d22f0 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.977+0000 7f887dff2700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780cee50 con 0x7f88780ce4d0 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.977+0000 7f887dff2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887815e460 con 0x7f887815e080 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f8877fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f88780d22f0 0x7f88780d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38706/0 (socket says 172.21.15.113:38706) 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f8877fff700 1 -- 172.21.15.113:0/992686357 learned_addr learned my addr 172.21.15.113:0/992686357 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88777fe700 1 --2- 172.21.15.113:0/992686357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f887815e080 0x7f88780cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88767fc700 1 -- 172.21.15.113:0/992686357 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679292024 0 0) 0x7f88780d0150 con 0x7f88780d22f0 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88767fc700 1 -- 172.21.15.113:0/992686357 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88600035b0 con 0x7f88780d22f0 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88767fc700 1 -- 172.21.15.113:0/992686357 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939547907 0 0) 0x7f88780cee50 con 0x7f88780ce4d0 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88777fe700 1 -- 172.21.15.113:0/992686357 >> v1:172.21.15.124:6789/0 conn(0x7f88780ce4d0 legacy=0x7f88780d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88777fe700 1 -- 172.21.15.113:0/992686357 >> v1:172.21.15.113:6789/0 conn(0x7f88780d22f0 legacy=0x7f88780d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88777fe700 1 -- 172.21.15.113:0/992686357 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780cf130 con 0x7f887815e080 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88777fe700 1 --2- 172.21.15.113:0/992686357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f887815e080 0x7f88780cc2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8868009870 tx=0x7f8868009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad3aa6a71640f0b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88767fc700 1 -- 172.21.15.113:0/992686357 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868011830 con 0x7f887815e080 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88767fc700 1 -- 172.21.15.113:0/992686357 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8868011990 con 0x7f887815e080 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.978+0000 7f88767fc700 1 -- 172.21.15.113:0/992686357 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868011d20 con 0x7f887815e080 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 -- 172.21.15.113:0/992686357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f887815e080 msgr2=0x7f88780cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 --2- 172.21.15.113:0/992686357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f887815e080 0x7f88780cc2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8868009870 tx=0x7f8868009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 -- 172.21.15.113:0/992686357 shutdown_connections 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 --2- 172.21.15.113:0/992686357 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f887815e080 0x7f88780cc2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 -- 172.21.15.113:0/992686357 >> 172.21.15.113:0/992686357 conn(0x7f8878073130 msgr2=0x7f887809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 -- 172.21.15.113:0/992686357 shutdown_connections 2024-01-16T13:52:34.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.979+0000 7f887dff2700 1 -- 172.21.15.113:0/992686357 wait complete. 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.980+0000 7f887dff2700 1 Processor -- start 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.980+0000 7f887dff2700 1 -- start start 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f887dff2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88780ce4d0 0x7f88780b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f887dff2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 0x7f887814fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f887dff2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f887815e080 0x7f88780ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f887dff2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88780b4ce0 con 0x7f88780d22f0 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f887dff2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88780b4fc0 con 0x7f887815e080 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f887dff2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88780b52a0 con 0x7f88780ce4d0 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f8876ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 0x7f887814fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f8876ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 0x7f887814fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55938/0 (socket says 172.21.15.113:55938) 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f8876ffd700 1 -- 172.21.15.113:0/3858447423 learned_addr learned my addr 172.21.15.113:0/3858447423 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f88777fe700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88780ce4d0 0x7f88780b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f8877fff700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f887815e080 0x7f88780ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.981+0000 7f8876ffd700 1 -- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88780ce4d0 msgr2=0x7f88780b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f8876ffd700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88780ce4d0 0x7f88780b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f8876ffd700 1 -- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f887815e080 msgr2=0x7f88780ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f8876ffd700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f887815e080 0x7f88780ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f8876ffd700 1 -- 172.21.15.113:0/3858447423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88680094a0 con 0x7f88780d22f0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f8877fff700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f887815e080 0x7f88780ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f8876ffd700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 0x7f887814fa50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f886c0011e0 tx=0x7f886c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0054e0 con 0x7f88780d22f0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c003680 con 0x7f88780d22f0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88780b5580 con 0x7f88780d22f0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c003a30 con 0x7f88780d22f0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.982+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88780b62c0 con 0x7f88780d22f0 2024-01-16T13:52:34.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.983+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88780cc790 con 0x7f88780d22f0 2024-01-16T13:52:34.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.983+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f886c003b90 con 0x7f88780d22f0 2024-01-16T13:52:34.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.983+0000 7f885ffff700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f886406b350 0x7f886406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:34.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.984+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f886c089050 con 0x7f88780d22f0 2024-01-16T13:52:34.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.984+0000 7f88777fe700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f886406b350 0x7f886406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:34.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.984+0000 7f88777fe700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f886406b350 0x7f886406d810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88680097c0 tx=0x7f886801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:34.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:34.988+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f886c053ed0 con 0x7f88780d22f0 2024-01-16T13:52:35.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.257+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f8878067f60 con 0x7f88780d22f0 2024-01-16T13:52:35.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.572+0000 7f885ffff700 1 -- 172.21.15.113:0/3858447423 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v49) v1 ==== 155+0+0 (secure 0 0 0) 0x7f886c0574f0 con 0x7f88780d22f0 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f886406b350 msgr2=0x7f886406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f886406b350 0x7f886406d810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88680097c0 tx=0x7f886801c040 comp rx=0 tx=0).stop 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 msgr2=0x7f887814fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 0x7f887814fa50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f886c0011e0 tx=0x7f886c00ca30 comp rx=0 tx=0).stop 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 shutdown_connections 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f886406b350 0x7f886406d810 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f887815e080 0x7f88780ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88780ce4d0 0x7f88780b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 --2- 172.21.15.113:0/3858447423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88780d22f0 0x7f887814fa50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 >> 172.21.15.113:0/3858447423 conn(0x7f8878073130 msgr2=0x7f887809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:35.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 shutdown_connections 2024-01-16T13:52:35.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.575+0000 7f887dff2700 1 -- 172.21.15.113:0/3858447423 wait complete. 2024-01-16T13:52:35.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T13:52:35.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.712+0000 7fca25c4b700 1 Processor -- start 2024-01-16T13:52:35.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.712+0000 7fca25c4b700 1 -- start start 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca25c4b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca25c4b700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca200d0790 con 0x7fca200ceb20 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca25c4b700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2006ee70 con 0x7fca200cf020 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca25c4b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2006f150 con 0x7fca2015bee0 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca1ffff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fca200ceb20 0x7fca2006f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38716/0 (socket says 172.21.15.113:38716) 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca1ffff700 1 -- 172.21.15.113:0/3367708402 learned_addr learned my addr 172.21.15.113:0/3367708402 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/3367708402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598347782 0 0) 0x7fca200d0790 con 0x7fca200ceb20 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.713+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0035b0 con 0x7fca200ceb20 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648207169 0 0) 0x7fca2006ee70 con 0x7fca200cf020 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca200d0790 con 0x7fca200cf020 2024-01-16T13:52:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1f7fe700 1 -- 172.21.15.113:0/3367708402 >> v1:172.21.15.124:6789/0 conn(0x7fca200cf020 legacy=0x7fca200ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1f7fe700 1 -- 172.21.15.113:0/3367708402 >> v1:172.21.15.113:6789/0 conn(0x7fca200ceb20 legacy=0x7fca2006f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815110684 0 0) 0x7fca0c0035b0 con 0x7fca200ceb20 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1f7fe700 1 -- 172.21.15.113:0/3367708402 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2006f430 con 0x7fca2015bee0 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/3367708402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200d01b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca08009870 tx=0x7fca08009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2351183234d33bfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca180019f0 con 0x7fca2015bee0 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca08004030 con 0x7fca2015bee0 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.714+0000 7fca1e7fc700 1 -- 172.21.15.113:0/3367708402 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca080043c0 con 0x7fca2015bee0 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 -- 172.21.15.113:0/3367708402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 msgr2=0x7fca200d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 --2- 172.21.15.113:0/3367708402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200d01b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca08009870 tx=0x7fca08009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 -- 172.21.15.113:0/3367708402 shutdown_connections 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 --2- 172.21.15.113:0/3367708402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200d01b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 -- 172.21.15.113:0/3367708402 >> 172.21.15.113:0/3367708402 conn(0x7fca200b3780 msgr2=0x7fca200b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 -- 172.21.15.113:0/3367708402 shutdown_connections 2024-01-16T13:52:35.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.715+0000 7fca25c4b700 1 -- 172.21.15.113:0/3367708402 wait complete. 2024-01-16T13:52:35.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.716+0000 7fca25c4b700 1 Processor -- start 2024-01-16T13:52:35.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.716+0000 7fca25c4b700 1 -- start start 2024-01-16T13:52:35.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca25c4b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca200ceb20 0x7fca200b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:35.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca25c4b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 0x7fca200b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:35.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca25c4b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca25c4b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2015c440 con 0x7fca200cf020 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca25c4b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2015dbd0 con 0x7fca200ceb20 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca25c4b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca200b6db0 con 0x7fca2015bee0 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 0x7fca200b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 0x7fca200b6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55942/0 (socket says 172.21.15.113:55942) 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 -- 172.21.15.113:0/1231916832 learned_addr learned my addr 172.21.15.113:0/1231916832 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca200ceb20 0x7fca200b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1ffff700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 -- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 msgr2=0x7fca200c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 -- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca200ceb20 msgr2=0x7fca200b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca200ceb20 0x7fca200b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 -- 172.21.15.113:0/1231916832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca080094a0 con 0x7fca200cf020 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca1effd700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 0x7fca200b6590 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c520 tx=0x7fca1000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca100052d0 con 0x7fca200cf020 2024-01-16T13:52:35.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca10005430 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.717+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca100038c0 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.718+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca200b7090 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.718+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca200b7650 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.718+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca20069020 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.719+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fca10003a20 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.719+0000 7fca17fff700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca0006b350 0x7fca0006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.719+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca10088f80 con 0x7fca200cf020 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.719+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca0006b350 0x7fca0006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:35.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.720+0000 7fca1f7fe700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca0006b350 0x7fca0006d810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca080090b0 tx=0x7fca08008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:35.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.724+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca100bbe90 con 0x7fca200cf020 2024-01-16T13:52:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.916+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fca200c89d0 con 0x7fca200cf020 2024-01-16T13:52:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.920+0000 7fca17fff700 1 -- 172.21.15.113:0/1231916832 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca100170c0 con 0x7fca200cf020 2024-01-16T13:52:35.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca0006b350 msgr2=0x7fca0006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca0006b350 0x7fca0006d810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fca080090b0 tx=0x7fca08008030 comp rx=0 tx=0).stop 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 msgr2=0x7fca200b6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 0x7fca200b6590 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c520 tx=0x7fca1000c830 comp rx=0 tx=0).stop 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 shutdown_connections 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fca0006b350 0x7fca0006d810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fca2015bee0 0x7fca200c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca200ceb20 0x7fca200b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 --2- 172.21.15.113:0/1231916832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca200cf020 0x7fca200b6590 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 >> 172.21.15.113:0/1231916832 conn(0x7fca200b3780 msgr2=0x7fca200b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 shutdown_connections 2024-01-16T13:52:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:35.922+0000 7fca25c4b700 1 -- 172.21.15.113:0/1231916832 wait complete. 2024-01-16T13:52:35.917 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T13:52:35.918 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T13:52:35.957 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T13:52:36.003 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T13:52:36.003 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:36.003 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:36.021 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:36.021 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:36.079 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:36.079 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:36.136 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:36.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.314+0000 7f7ba809c700 1 Processor -- start 2024-01-16T13:52:36.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.315+0000 7f7ba809c700 1 -- start start 2024-01-16T13:52:36.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.315+0000 7f7ba809c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ba01446a0 0x7f7ba00b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.315+0000 7f7ba809c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7ba00adf00 0x7f7ba006c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba809c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba006c540 0x7f7ba006da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba809c700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba0144a80 con 0x7f7ba00adf00 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba809c700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba006c9c0 con 0x7f7ba01446a0 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba809c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba00af950 con 0x7f7ba006c540 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7ba00adf00 0x7f7ba006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7ba00adf00 0x7f7ba006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55950/0 (socket says 172.21.15.113:55950) 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 -- 172.21.15.113:0/374786019 learned_addr learned my addr 172.21.15.113:0/374786019 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba6639700 1 --2- 172.21.15.113:0/374786019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba006c540 0x7f7ba006da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5e38700 1 --2- 172.21.15.113:0/374786019 >> v2:172.21.15.124:3300/0 conn(0x7f7ba01446a0 0x7f7ba00b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 -- 172.21.15.113:0/374786019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba006c540 msgr2=0x7f7ba006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 --2- 172.21.15.113:0/374786019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba006c540 0x7f7ba006da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 -- 172.21.15.113:0/374786019 >> v2:172.21.15.124:3300/0 conn(0x7f7ba01446a0 msgr2=0x7f7ba00b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 --2- 172.21.15.113:0/374786019 >> v2:172.21.15.124:3300/0 conn(0x7f7ba01446a0 0x7f7ba00b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 -- 172.21.15.113:0/374786019 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba00afc30 con 0x7f7ba00adf00 2024-01-16T13:52:36.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba5637700 1 --2- 172.21.15.113:0/374786019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba006c000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00729a0 tx=0x7f7b94009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307e2d3e95a1065 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba4e36700 1 -- 172.21.15.113:0/374786019 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9400b870 con 0x7f7ba00adf00 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba4e36700 1 -- 172.21.15.113:0/374786019 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9400b9d0 con 0x7f7ba00adf00 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.316+0000 7f7ba4e36700 1 -- 172.21.15.113:0/374786019 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9400bd60 con 0x7f7ba00adf00 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 -- 172.21.15.113:0/374786019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 msgr2=0x7f7ba006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 --2- 172.21.15.113:0/374786019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba006c000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00729a0 tx=0x7f7b94009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 -- 172.21.15.113:0/374786019 shutdown_connections 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 --2- 172.21.15.113:0/374786019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba006c540 0x7f7ba006da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 --2- 172.21.15.113:0/374786019 >> v2:172.21.15.124:3300/0 conn(0x7f7ba01446a0 0x7f7ba00b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 --2- 172.21.15.113:0/374786019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba006c000 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 -- 172.21.15.113:0/374786019 >> 172.21.15.113:0/374786019 conn(0x7f7ba0071b90 msgr2=0x7f7ba0071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 -- 172.21.15.113:0/374786019 shutdown_connections 2024-01-16T13:52:36.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.317+0000 7f7ba809c700 1 -- 172.21.15.113:0/374786019 wait complete. 2024-01-16T13:52:36.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.318+0000 7f7ba809c700 1 Processor -- start 2024-01-16T13:52:36.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.318+0000 7f7ba809c700 1 -- start start 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba809c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba006c540 0x7f7ba015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba809c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba0136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba809c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba01446a0 0x7f7ba00cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba809c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba01374e0 con 0x7f7ba00adf00 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba809c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba00d0440 con 0x7f7ba006c540 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba809c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba00ce260 con 0x7f7ba01446a0 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba0136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba0136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55960/0 (socket says 172.21.15.113:55960) 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 -- 172.21.15.113:0/2425535979 learned_addr learned my addr 172.21.15.113:0/2425535979 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5e38700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba006c540 0x7f7ba015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 -- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba01446a0 msgr2=0x7f7ba00cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba6639700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba01446a0 0x7f7ba00cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba01446a0 0x7f7ba00cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 -- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba006c540 msgr2=0x7f7ba015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba006c540 0x7f7ba015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 -- 172.21.15.113:0/2425535979 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b94009550 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5e38700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba006c540 0x7f7ba015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba6639700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba01446a0 0x7f7ba00cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.319+0000 7f7ba5637700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba0136fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7b94000c00 tx=0x7f7b94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.320+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94004540 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.320+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b940036a0 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.320+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba00d0160 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.320+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94003a50 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.320+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba0137950 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.321+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba00753f0 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.321+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b94021070 con 0x7f7ba00adf00 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.321+0000 7f7b927fc700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7b8406b350 0x7f7b8406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.321+0000 7f7ba5e38700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7b8406b350 0x7f7b8406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.321+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b9408b250 con 0x7f7ba00adf00 2024-01-16T13:52:36.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.322+0000 7f7ba5e38700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7b8406b350 0x7f7b8406d810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0140870 tx=0x7f7b9c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.326+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b94056150 con 0x7f7ba00adf00 2024-01-16T13:52:36.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.518+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ba0067f60 con 0x7f7ba00adf00 2024-01-16T13:52:36.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.519+0000 7f7b927fc700 1 -- 172.21.15.113:0/2425535979 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f7b94059770 con 0x7f7ba00adf00 2024-01-16T13:52:36.503 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:36.503 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7b8406b350 msgr2=0x7f7b8406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7b8406b350 0x7f7b8406d810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0140870 tx=0x7f7b9c008040 comp rx=0 tx=0).stop 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 msgr2=0x7f7ba0136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba0136fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7b94000c00 tx=0x7f7b94004750 comp rx=0 tx=0).stop 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 shutdown_connections 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f7b8406b350 0x7f7b8406d810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7ba01446a0 0x7f7ba00cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba006c540 0x7f7ba015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 --2- 172.21.15.113:0/2425535979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ba00adf00 0x7f7ba0136fa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 >> 172.21.15.113:0/2425535979 conn(0x7f7ba0071b90 msgr2=0x7f7ba015e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:36.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 shutdown_connections 2024-01-16T13:52:36.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.522+0000 7f7ba809c700 1 -- 172.21.15.113:0/2425535979 wait complete. 2024-01-16T13:52:36.517 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:36.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:36.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.659+0000 7ff3a4388700 1 Processor -- start 2024-01-16T13:52:36.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.659+0000 7ff3a4388700 1 -- start start 2024-01-16T13:52:36.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a4388700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff39c161690 0x7ff39c0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a4388700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a4388700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c0d0a70 con 0x7ff39c0d1730 2024-01-16T13:52:36.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a4388700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39c0d1c10 con 0x7ff39c161690 2024-01-16T13:52:36.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a4388700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c0b2ab0 con 0x7ff39c161a70 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2124700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff39c161690 0x7ff39c0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2925700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff39c0d1730 0x7ff39c09aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38728/0 (socket says 172.21.15.113:38728) 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2925700 1 -- 172.21.15.113:0/3585602844 learned_addr learned my addr 172.21.15.113:0/3585602844 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a1923700 1 --2- 172.21.15.113:0/3585602844 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2124700 1 -- 172.21.15.113:0/3585602844 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 msgr2=0x7ff39c0d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2124700 1 --2- 172.21.15.113:0/3585602844 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2124700 1 -- 172.21.15.113:0/3585602844 >> v1:172.21.15.113:6789/0 conn(0x7ff39c0d1730 legacy=0x7ff39c09aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2124700 1 -- 172.21.15.113:0/3585602844 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c0b2d90 con 0x7ff39c161690 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a1122700 1 -- 172.21.15.113:0/3585602844 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813270452 0 0) 0x7ff39c0d0a70 con 0x7ff39c0d1730 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a2124700 1 --2- 172.21.15.113:0/3585602844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c0d0f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff398009840 tx=0x7ff398009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc78c4bec79a7152 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a1122700 1 -- 172.21.15.113:0/3585602844 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398011830 con 0x7ff39c161690 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a1122700 1 -- 172.21.15.113:0/3585602844 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398011990 con 0x7ff39c161690 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.660+0000 7ff3a1122700 1 -- 172.21.15.113:0/3585602844 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398011d20 con 0x7ff39c161690 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 -- 172.21.15.113:0/3585602844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 msgr2=0x7ff39c0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3585602844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c0d0f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff398009840 tx=0x7ff398009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 -- 172.21.15.113:0/3585602844 shutdown_connections 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3585602844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c0d0f20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3585602844 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 -- 172.21.15.113:0/3585602844 >> 172.21.15.113:0/3585602844 conn(0x7ff39c0d1e50 msgr2=0x7ff39c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 -- 172.21.15.113:0/3585602844 shutdown_connections 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.661+0000 7ff3a4388700 1 -- 172.21.15.113:0/3585602844 wait complete. 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.662+0000 7ff3a4388700 1 Processor -- start 2024-01-16T13:52:36.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.662+0000 7ff3a4388700 1 -- start start 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a4388700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff39c0d1730 0x7ff39c1f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a4388700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c1f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a4388700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a4388700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c14bdf0 con 0x7ff39c161690 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a4388700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c1f4ba0 con 0x7ff39c0d1730 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a4388700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c1f3a50 con 0x7ff39c161a70 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c1f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c1f3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55976/0 (socket says 172.21.15.113:55976) 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 -- 172.21.15.113:0/3587190036 learned_addr learned my addr 172.21.15.113:0/3587190036 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a2925700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a2124700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff39c0d1730 0x7ff39c1f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 -- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 msgr2=0x7ff39c0b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 -- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff39c0d1730 msgr2=0x7ff39c1f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff39c0d1730 0x7ff39c1f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.663+0000 7ff3a1923700 1 -- 172.21.15.113:0/3587190036 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980094a0 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.664+0000 7ff3a1923700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c1f3510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00e980 tx=0x7ff38c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.664+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c009710 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.664+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c00c690 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.664+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c0af210 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.664+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c00ca40 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.664+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c0af7d0 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.665+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c0c5af0 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.665+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff38c00cba0 con 0x7ff39c161690 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.665+0000 7ff392ffd700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff38006b350 0x7ff38006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.666+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff38c089bf0 con 0x7ff39c161690 2024-01-16T13:52:36.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.666+0000 7ff3a2124700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff38006b350 0x7ff38006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.666+0000 7ff3a2124700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff38006b350 0x7ff38006d810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff398009fd0 tx=0x7ff39801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.670+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff39c0c5af0 con 0x7ff39c161690 2024-01-16T13:52:36.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.859+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff39c0c5af0 con 0x7ff39c161690 2024-01-16T13:52:36.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.859+0000 7ff392ffd700 1 -- 172.21.15.113:0/3587190036 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff39c0c5af0 con 0x7ff39c161690 2024-01-16T13:52:36.842 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff38006b350 msgr2=0x7ff38006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff38006b350 0x7ff38006d810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff398009fd0 tx=0x7ff39801c040 comp rx=0 tx=0).stop 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 msgr2=0x7ff39c1f3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c1f3510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00e980 tx=0x7ff38c00ec90 comp rx=0 tx=0).stop 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 shutdown_connections 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff38006b350 0x7ff38006d810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff39c161a70 0x7ff39c0b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff39c0d1730 0x7ff39c1f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 --2- 172.21.15.113:0/3587190036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff39c161690 0x7ff39c1f3510 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 >> 172.21.15.113:0/3587190036 conn(0x7ff39c0d1e50 msgr2=0x7ff39c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 shutdown_connections 2024-01-16T13:52:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:36.861+0000 7ff3a4388700 1 -- 172.21.15.113:0/3587190036 wait complete. 2024-01-16T13:52:36.856 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:36.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 Processor -- start 2024-01-16T13:52:36.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 -- start start 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcd8c15e1e0 0x7fcd8c0d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b3050 0x7fcd8c0d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c0d30b0 con 0x7fcd8c0b2b00 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8c0ce730 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.000+0000 7fcd919b3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c162920 con 0x7fcd8c0b3050 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcd8c15e1e0 0x7fcd8c0d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcd8c15e1e0 0x7fcd8c0d0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55990/0 (socket says 172.21.15.113:55990) 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 -- 172.21.15.113:0/1989967759 learned_addr learned my addr 172.21.15.113:0/1989967759 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8b7fe700 1 --2- 172.21.15.113:0/1989967759 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b3050 0x7fcd8c0d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 -- 172.21.15.113:0/1989967759 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b3050 msgr2=0x7fcd8c0d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 --2- 172.21.15.113:0/1989967759 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b3050 0x7fcd8c0d11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 -- 172.21.15.113:0/1989967759 >> v1:172.21.15.124:6789/0 conn(0x7fcd8c0b2b00 legacy=0x7fcd8c0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 -- 172.21.15.113:0/1989967759 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c0ce940 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd8affd700 1 --2- 172.21.15.113:0/1989967759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c0d0c80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0098f0 tx=0x7fcd7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505c839a1217318b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd89ffb700 1 -- 172.21.15.113:0/1989967759 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c004230 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd89ffb700 1 -- 172.21.15.113:0/1989967759 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c004390 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.001+0000 7fcd89ffb700 1 -- 172.21.15.113:0/1989967759 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c0036a0 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 -- 172.21.15.113:0/1989967759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 msgr2=0x7fcd8c0d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 --2- 172.21.15.113:0/1989967759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c0d0c80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0098f0 tx=0x7fcd7c009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 -- 172.21.15.113:0/1989967759 shutdown_connections 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 --2- 172.21.15.113:0/1989967759 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b3050 0x7fcd8c0d11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 --2- 172.21.15.113:0/1989967759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c0d0c80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 -- 172.21.15.113:0/1989967759 >> 172.21.15.113:0/1989967759 conn(0x7fcd8c0717f0 msgr2=0x7fcd8c071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:36.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 -- 172.21.15.113:0/1989967759 shutdown_connections 2024-01-16T13:52:36.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.002+0000 7fcd919b3700 1 -- 172.21.15.113:0/1989967759 wait complete. 2024-01-16T13:52:36.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.003+0000 7fcd919b3700 1 Processor -- start 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.003+0000 7fcd919b3700 1 -- start start 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd919b3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b2b00 0x7fcd8c0cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd919b3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcd8c0b3050 0x7fcd8c1f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd919b3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd919b3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0b4df0 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd919b3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c1f8570 con 0x7fcd8c0b3050 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd919b3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c1f8850 con 0x7fcd8c0b2b00 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c15e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56000/0 (socket says 172.21.15.113:56000) 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 -- 172.21.15.113:0/3022982629 learned_addr learned my addr 172.21.15.113:0/3022982629 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8affd700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b2b00 0x7fcd8c0cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8a7fc700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcd8c0b3050 0x7fcd8c1f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 -- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b2b00 msgr2=0x7fcd8c0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b2b00 0x7fcd8c0cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 -- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcd8c0b3050 msgr2=0x7fcd8c1f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcd8c0b3050 0x7fcd8c1f7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 -- 172.21.15.113:0/3022982629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c009550 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd8b7fe700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c15e5c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400cc30 tx=0x7fcd8400a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400ab70 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.004+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8400acd0 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.005+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c0b4a80 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.005+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84003c40 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.005+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c0b6540 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.005+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c0b4fa0 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.006+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcd8400f3f0 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.006+0000 7fcd73fff700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcd7806b350 0x7fcd7806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.006+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd84089dc0 con 0x7fcd8c15e1e0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.006+0000 7fcd8affd700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcd7806b350 0x7fcd7806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:36.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.007+0000 7fcd8affd700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcd7806b350 0x7fcd7806d810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c1609b0 tx=0x7fcd7c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:36.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.011+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd8c0b4fa0 con 0x7fcd8c15e1e0 2024-01-16T13:52:37.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.199+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd8c1f8a10 con 0x7fcd8c15e1e0 2024-01-16T13:52:37.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.199+0000 7fcd73fff700 1 -- 172.21.15.113:0/3022982629 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd8401d020 con 0x7fcd8c15e1e0 2024-01-16T13:52:37.182 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcd7806b350 msgr2=0x7fcd7806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcd7806b350 0x7fcd7806d810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c1609b0 tx=0x7fcd7c008030 comp rx=0 tx=0).stop 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 msgr2=0x7fcd8c15e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c15e5c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400cc30 tx=0x7fcd8400a1f0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 shutdown_connections 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcd7806b350 0x7fcd7806d810 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd8c15e1e0 0x7fcd8c15e5c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcd8c0b2b00 0x7fcd8c0cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 --2- 172.21.15.113:0/3022982629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcd8c0b3050 0x7fcd8c1f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.201+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 >> 172.21.15.113:0/3022982629 conn(0x7fcd8c0717f0 msgr2=0x7fcd8c0b8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:37.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.202+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 shutdown_connections 2024-01-16T13:52:37.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.202+0000 7fcd919b3700 1 -- 172.21.15.113:0/3022982629 wait complete. 2024-01-16T13:52:37.196 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:37.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.338+0000 7f68a7dbd700 1 Processor -- start 2024-01-16T13:52:37.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.338+0000 7f68a7dbd700 1 -- start start 2024-01-16T13:52:37.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a7dbd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68a0161690 0x7f68a00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a7dbd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f68a009b110 0x7f68a00d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a7dbd700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00d0a70 con 0x7f68a0161a70 2024-01-16T13:52:37.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a7dbd700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a00d1c10 con 0x7f68a0161690 2024-01-16T13:52:37.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a7dbd700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a00b2ab0 con 0x7f68a009b110 2024-01-16T13:52:37.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5b59700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68a0161690 0x7f68a00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5358700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f68a0161a70 0x7f68a009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38734/0 (socket says 172.21.15.113:38734) 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5358700 1 -- 172.21.15.113:0/1850733366 learned_addr learned my addr 172.21.15.113:0/1850733366 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a635a700 1 --2- 172.21.15.113:0/1850733366 >> v2:172.21.15.124:3300/0 conn(0x7f68a009b110 0x7f68a00d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5b59700 1 -- 172.21.15.113:0/1850733366 >> v2:172.21.15.124:3300/0 conn(0x7f68a009b110 msgr2=0x7f68a00d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5b59700 1 --2- 172.21.15.113:0/1850733366 >> v2:172.21.15.124:3300/0 conn(0x7f68a009b110 0x7f68a00d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5b59700 1 -- 172.21.15.113:0/1850733366 >> v1:172.21.15.113:6789/0 conn(0x7f68a0161a70 legacy=0x7f68a009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5b59700 1 -- 172.21.15.113:0/1850733366 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a00b2d90 con 0x7f68a0161690 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a4b57700 1 -- 172.21.15.113:0/1850733366 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467011539 0 0) 0x7f68a00d0a70 con 0x7f68a0161a70 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a5b59700 1 --2- 172.21.15.113:0/1850733366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a0161690 0x7f68a00d0f20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f68a00ce460 tx=0x7f6894009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbe9ccad757f524 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a4b57700 1 -- 172.21.15.113:0/1850733366 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894011830 con 0x7f68a0161690 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a4b57700 1 -- 172.21.15.113:0/1850733366 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6894011990 con 0x7f68a0161690 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.339+0000 7f68a4b57700 1 -- 172.21.15.113:0/1850733366 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894011d20 con 0x7f68a0161690 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1850733366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a0161690 msgr2=0x7f68a00d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1850733366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a0161690 0x7f68a00d0f20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f68a00ce460 tx=0x7f6894009b20 comp rx=0 tx=0).stop 2024-01-16T13:52:37.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1850733366 shutdown_connections 2024-01-16T13:52:37.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1850733366 >> v2:172.21.15.124:3300/0 conn(0x7f68a009b110 0x7f68a00d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1850733366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a0161690 0x7f68a00d0f20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1850733366 >> 172.21.15.113:0/1850733366 conn(0x7f68a00d1e50 msgr2=0x7f68a00d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:37.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1850733366 shutdown_connections 2024-01-16T13:52:37.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.340+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1850733366 wait complete. 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.341+0000 7f68a7dbd700 1 Processor -- start 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.342+0000 7f68a7dbd700 1 -- start start 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.342+0000 7f68a7dbd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 0x7f68a0158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68a0161690 0x7f68a00b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f68a0161a70 0x7f68a00b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a00b0920 con 0x7f68a009b110 2024-01-16T13:52:37.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a00b5df0 con 0x7f68a0161690 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a00afca0 con 0x7f68a0161a70 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 0x7f68a0158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 0x7f68a0158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56020/0 (socket says 172.21.15.113:56020) 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 -- 172.21.15.113:0/1674814823 learned_addr learned my addr 172.21.15.113:0/1674814823 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5358700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68a0161690 0x7f68a00b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a635a700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f68a0161a70 0x7f68a00b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 -- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f68a0161a70 msgr2=0x7f68a00b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f68a0161a70 0x7f68a00b58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 -- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68a0161690 msgr2=0x7f68a00b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68a0161690 0x7f68a00b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 -- 172.21.15.113:0/1674814823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68940094a0 con 0x7f68a009b110 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a5b59700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 0x7f68a0158020 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68940090b0 tx=0x7f6894004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68940038a0 con 0x7f68a009b110 2024-01-16T13:52:37.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689401e070 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a00aff80 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894003c20 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.343+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a00b0540 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.344+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a00b0a60 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.345+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f689401a420 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.345+0000 7f68927fc700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f688406b350 0x7f688406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.345+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f689408aba0 con 0x7f68a009b110 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.345+0000 7f68a5358700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f688406b350 0x7f688406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.345+0000 7f68a5358700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f688406b350 0x7f688406d810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f68a00b7cd0 tx=0x7f689c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:37.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.349+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6894055aa0 con 0x7f68a009b110 2024-01-16T13:52:37.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.540+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f68a0069020 con 0x7f68a009b110 2024-01-16T13:52:37.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.540+0000 7f68927fc700 1 -- 172.21.15.113:0/1674814823 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f68940150a0 con 0x7f68a009b110 2024-01-16T13:52:37.523 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:37.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f688406b350 msgr2=0x7f688406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f688406b350 0x7f688406d810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f68a00b7cd0 tx=0x7f689c009040 comp rx=0 tx=0).stop 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 msgr2=0x7f68a0158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 0x7f68a0158020 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68940090b0 tx=0x7f6894004750 comp rx=0 tx=0).stop 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 shutdown_connections 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f688406b350 0x7f688406d810 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f68a0161a70 0x7f68a00b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68a009b110 0x7f68a0158020 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 --2- 172.21.15.113:0/1674814823 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68a0161690 0x7f68a00b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 >> 172.21.15.113:0/1674814823 conn(0x7f68a00d1e50 msgr2=0x7f68a0070790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 shutdown_connections 2024-01-16T13:52:37.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.542+0000 7f68a7dbd700 1 -- 172.21.15.113:0/1674814823 wait complete. 2024-01-16T13:52:37.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T13:52:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.678+0000 7fdc397dd700 1 Processor -- start 2024-01-16T13:52:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc397dd700 1 -- start start 2024-01-16T13:52:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc397dd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdc3415bed0 0x7fdc340b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc397dd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdc341603c0 0x7fdc340d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc397dd700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc340d0770 con 0x7fdc340d1a50 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc397dd700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3415c2b0 con 0x7fdc341603c0 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc397dd700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc340cd100 con 0x7fdc3415bed0 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.679+0000 7fdc327fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdc341603c0 0x7fdc340d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdc341603c0 0x7fdc340d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56036/0 (socket says 172.21.15.113:56036) 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 -- 172.21.15.113:0/358839787 learned_addr learned my addr 172.21.15.113:0/358839787 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc32ffd700 1 --2- 172.21.15.113:0/358839787 >> v2:172.21.15.124:3300/0 conn(0x7fdc3415bed0 0x7fdc340b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 -- 172.21.15.113:0/358839787 >> v2:172.21.15.124:3300/0 conn(0x7fdc3415bed0 msgr2=0x7fdc340b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 --2- 172.21.15.113:0/358839787 >> v2:172.21.15.124:3300/0 conn(0x7fdc3415bed0 0x7fdc340b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 -- 172.21.15.113:0/358839787 >> v1:172.21.15.124:6789/0 conn(0x7fdc340d1a50 legacy=0x7fdc340d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 -- 172.21.15.113:0/358839787 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340d09a0 con 0x7fdc341603c0 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc32ffd700 1 --2- 172.21.15.113:0/358839787 >> v2:172.21.15.124:3300/0 conn(0x7fdc3415bed0 0x7fdc340b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc327fc700 1 --2- 172.21.15.113:0/358839787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc341603c0 0x7fdc340d0230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdc3409b2a0 tx=0x7fdc24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661fa9ea391b8ca3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc31ffb700 1 -- 172.21.15.113:0/358839787 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24004230 con 0x7fdc341603c0 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc31ffb700 1 -- 172.21.15.113:0/358839787 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc24004390 con 0x7fdc341603c0 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.680+0000 7fdc31ffb700 1 -- 172.21.15.113:0/358839787 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc240036a0 con 0x7fdc341603c0 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 -- 172.21.15.113:0/358839787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc341603c0 msgr2=0x7fdc340d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 --2- 172.21.15.113:0/358839787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc341603c0 0x7fdc340d0230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdc3409b2a0 tx=0x7fdc24009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 -- 172.21.15.113:0/358839787 shutdown_connections 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 --2- 172.21.15.113:0/358839787 >> v2:172.21.15.124:3300/0 conn(0x7fdc3415bed0 0x7fdc340b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 --2- 172.21.15.113:0/358839787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc341603c0 0x7fdc340d0230 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 -- 172.21.15.113:0/358839787 >> 172.21.15.113:0/358839787 conn(0x7fdc34073130 msgr2=0x7fdc3409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 -- 172.21.15.113:0/358839787 shutdown_connections 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.681+0000 7fdc397dd700 1 -- 172.21.15.113:0/358839787 wait complete. 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.682+0000 7fdc397dd700 1 Processor -- start 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.682+0000 7fdc397dd700 1 -- start start 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc397dd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc340d1a50 0x7fdc340c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc397dd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 0x7fdc341f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc397dd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc341603c0 0x7fdc340cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc397dd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340b5f10 con 0x7fdc3415bed0 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc397dd700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341f17b0 con 0x7fdc340d1a50 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc397dd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341f1a10 con 0x7fdc341603c0 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 0x7fdc341f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 0x7fdc341f55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56038/0 (socket says 172.21.15.113:56038) 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 -- 172.21.15.113:0/2855562801 learned_addr learned my addr 172.21.15.113:0/2855562801 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc32ffd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc340d1a50 0x7fdc340c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc337fe700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc341603c0 0x7fdc340cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 -- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc341603c0 msgr2=0x7fdc340cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc341603c0 0x7fdc340cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 -- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc340d1a50 msgr2=0x7fdc340c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc340d1a50 0x7fdc340c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 -- 172.21.15.113:0/2855562801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc24009550 con 0x7fdc3415bed0 2024-01-16T13:52:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc32ffd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc340d1a50 0x7fdc340c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc327fc700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 0x7fdc341f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009130 tx=0x7fdc24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.683+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24011630 con 0x7fdc3415bed0 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.684+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc24011790 con 0x7fdc3415bed0 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.684+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc341f1cf0 con 0x7fdc3415bed0 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.684+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24011b40 con 0x7fdc3415bed0 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.684+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc341f22b0 con 0x7fdc3415bed0 2024-01-16T13:52:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.684+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc34067f60 con 0x7fdc3415bed0 2024-01-16T13:52:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.685+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdc24011ca0 con 0x7fdc3415bed0 2024-01-16T13:52:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.685+0000 7fdc1b7fe700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc2006b350 0x7fdc2006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.685+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc2408b1c0 con 0x7fdc3415bed0 2024-01-16T13:52:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.685+0000 7fdc32ffd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc2006b350 0x7fdc2006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.686+0000 7fdc32ffd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc2006b350 0x7fdc2006d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdc28006fd0 tx=0x7fdc28008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.690+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc240560c0 con 0x7fdc3415bed0 2024-01-16T13:52:37.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.895+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdc340b7bd0 con 0x7fdc3415bed0 2024-01-16T13:52:37.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.895+0000 7fdc1b7fe700 1 -- 172.21.15.113:0/2855562801 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdc240596e0 con 0x7fdc3415bed0 2024-01-16T13:52:37.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc2006b350 msgr2=0x7fdc2006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc2006b350 0x7fdc2006d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdc28006fd0 tx=0x7fdc28008040 comp rx=0 tx=0).stop 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 msgr2=0x7fdc341f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 0x7fdc341f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009130 tx=0x7fdc24004750 comp rx=0 tx=0).stop 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 shutdown_connections 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc2006b350 0x7fdc2006d810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc341603c0 0x7fdc340cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc340d1a50 0x7fdc340c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 --2- 172.21.15.113:0/2855562801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3415bed0 0x7fdc341f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 >> 172.21.15.113:0/2855562801 conn(0x7fdc34073130 msgr2=0x7fdc340b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 shutdown_connections 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:37.897+0000 7fdc397dd700 1 -- 172.21.15.113:0/2855562801 wait complete. 2024-01-16T13:52:37.881 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-16T13:52:37.892 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-16T13:52:37.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:37.893 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-01-16T13:52:37.935 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:52:37.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:37.935 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T13:52:38.011 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:38.011 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:38.032 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:38.032 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:38.094 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:38.094 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:38.154 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:38.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.292+0000 7fe365f73700 1 Processor -- start 2024-01-16T13:52:38.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.292+0000 7fe365f73700 1 -- start start 2024-01-16T13:52:38.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe365f73700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe360140f20 0x7fe3600b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe365f73700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe3600b2760 0x7fe36015e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe365f73700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe360161b40 con 0x7fe3600b3900 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe365f73700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3600b3e10 con 0x7fe3600b2760 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe365f73700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe360161d90 con 0x7fe360140f20 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe35effd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe3600b3900 0x7fe3600b2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38744/0 (socket says 172.21.15.113:38744) 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe35effd700 1 -- 172.21.15.113:0/2176952925 learned_addr learned my addr 172.21.15.113:0/2176952925 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414073848 0 0) 0x7fe360161b40 con 0x7fe3600b3900 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe34c0035b0 con 0x7fe3600b3900 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe35f7fe700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe360140f20 0x7fe3600b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.293+0000 7fe35ffff700 1 --2- 172.21.15.113:0/2176952925 >> v2:172.21.15.124:3300/0 conn(0x7fe3600b2760 0x7fe36015e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581815375 0 0) 0x7fe34c0035b0 con 0x7fe3600b3900 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe360140f20 msgr2=0x7fe3600b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe360140f20 0x7fe3600b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 >> v2:172.21.15.124:3300/0 conn(0x7fe3600b2760 msgr2=0x7fe36015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:38.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 --2- 172.21.15.113:0/2176952925 >> v2:172.21.15.124:3300/0 conn(0x7fe3600b2760 0x7fe36015e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe360162070 con 0x7fe3600b3900 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fe350001670 con 0x7fe3600b3900 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 >> v1:172.21.15.113:6789/0 conn(0x7fe3600b3900 legacy=0x7fe3600b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34c003a00 0x7fe34c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34c004850 0x7fe34c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c005210 0x7fe34c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600b3e10 con 0x7fe34c005210 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360161d90 con 0x7fe34c003a00 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c00c0b0 con 0x7fe34c004850 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35f7fe700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe360140f20 0x7fe3600b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 --2- 172.21.15.113:0/2176952925 >> v2:172.21.15.124:3300/0 conn(0x7fe3600b2760 0x7fe36015e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c005210 0x7fe34c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35effd700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34c004850 0x7fe34c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35f7fe700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34c003a00 0x7fe34c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 -- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34c004850 msgr2=0x7fe34c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34c004850 0x7fe34c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 -- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34c003a00 msgr2=0x7fe34c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34c003a00 0x7fe34c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 -- 172.21.15.113:0/2176952925 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c0035b0 con 0x7fe34c005210 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35ffff700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c005210 0x7fe34c00ba10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe3600d0190 tx=0x7fe34c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723195b7e1f42c91 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358007ea0 con 0x7fe34c005210 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358014400 con 0x7fe34c005210 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.294+0000 7fe35e7fc700 1 -- 172.21.15.113:0/2176952925 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358014770 con 0x7fe34c005210 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe365f73700 1 -- 172.21.15.113:0/2176952925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c005210 msgr2=0x7fe34c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe365f73700 1 --2- 172.21.15.113:0/2176952925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c005210 0x7fe34c00ba10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe3600d0190 tx=0x7fe34c005880 comp rx=0 tx=0).stop 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe35ffff700 1 -- 172.21.15.113:0/2176952925 reap_dead start 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe365f73700 1 -- 172.21.15.113:0/2176952925 shutdown_connections 2024-01-16T13:52:38.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe365f73700 1 -- 172.21.15.113:0/2176952925 >> 172.21.15.113:0/2176952925 conn(0x7fe3600743b0 msgr2=0x7fe36006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:38.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe365f73700 1 -- 172.21.15.113:0/2176952925 shutdown_connections 2024-01-16T13:52:38.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.295+0000 7fe365f73700 1 -- 172.21.15.113:0/2176952925 wait complete. 2024-01-16T13:52:38.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.296+0000 7fe365f73700 1 Processor -- start 2024-01-16T13:52:38.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.296+0000 7fe365f73700 1 -- start start 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe365f73700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe36015d9f0 0x7fe3601f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe365f73700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3600b15b0 0x7fe360160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe365f73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 0x7fe3601426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe365f73700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600cd8c0 con 0x7fe3601605b0 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe365f73700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360160a60 con 0x7fe36015d9f0 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe365f73700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe360144eb0 con 0x7fe3600b15b0 2024-01-16T13:52:38.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe35ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 0x7fe3601426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe35ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 0x7fe3601426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56058/0 (socket says 172.21.15.113:56058) 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe35ffff700 1 -- 172.21.15.113:0/1593415224 learned_addr learned my addr 172.21.15.113:0/1593415224 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe35effd700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3600b15b0 0x7fe360160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.297+0000 7fe35f7fe700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe36015d9f0 0x7fe3601f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe35ffff700 1 -- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3600b15b0 msgr2=0x7fe360160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe35ffff700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3600b15b0 0x7fe360160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe35ffff700 1 -- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe36015d9f0 msgr2=0x7fe3601f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe35ffff700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe36015d9f0 0x7fe3601f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe35ffff700 1 -- 172.21.15.113:0/1593415224 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3601f8b80 con 0x7fe3601605b0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe35ffff700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 0x7fe3601426b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe358014000 tx=0x7fe358004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358009690 con 0x7fe3601605b0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35801e610 con 0x7fe3601605b0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3600ccf20 con 0x7fe3601605b0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358007000 con 0x7fe3601605b0 2024-01-16T13:52:38.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.298+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3600cd4e0 con 0x7fe3601605b0 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.299+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3600cda00 con 0x7fe3601605b0 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.299+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe35801b050 con 0x7fe3601605b0 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.300+0000 7fe357fff700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe34006b350 0x7fe34006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.300+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe35808ace0 con 0x7fe3601605b0 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.300+0000 7fe35f7fe700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe34006b350 0x7fe34006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.300+0000 7fe35f7fe700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe34006b350 0x7fe34006d810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe348006f10 tx=0x7fe348008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.304+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3600cda00 con 0x7fe3601605b0 2024-01-16T13:52:38.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.496+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3601f8b80 con 0x7fe3601605b0 2024-01-16T13:52:38.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.497+0000 7fe357fff700 1 -- 172.21.15.113:0/1593415224 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fe3601f8b80 con 0x7fe3601605b0 2024-01-16T13:52:38.480 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:38.480 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:38.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe34006b350 msgr2=0x7fe34006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe34006b350 0x7fe34006d810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe348006f10 tx=0x7fe348008040 comp rx=0 tx=0).stop 2024-01-16T13:52:38.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 msgr2=0x7fe3601426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 0x7fe3601426b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe358014000 tx=0x7fe358004b40 comp rx=0 tx=0).stop 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 shutdown_connections 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe34006b350 0x7fe34006d810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3601605b0 0x7fe3601426b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe36015d9f0 0x7fe3601f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 --2- 172.21.15.113:0/1593415224 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3600b15b0 0x7fe360160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 >> 172.21.15.113:0/1593415224 conn(0x7fe3600743b0 msgr2=0x7fe36013f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 shutdown_connections 2024-01-16T13:52:38.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.499+0000 7fe365f73700 1 -- 172.21.15.113:0/1593415224 wait complete. 2024-01-16T13:52:38.494 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:38.495 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.637+0000 7fb28cf28700 1 Processor -- start 2024-01-16T13:52:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.637+0000 7fb28cf28700 1 -- start start 2024-01-16T13:52:38.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.638+0000 7fb28cf28700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 0x7fb2880b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.638+0000 7fb28cf28700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 0x7fb2880b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.638+0000 7fb28cf28700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2880b42a0 con 0x7fb2880b5050 2024-01-16T13:52:38.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.638+0000 7fb28cf28700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2880b5500 con 0x7fb28815e280 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.638+0000 7fb28cf28700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2880af900 con 0x7fb288073f90 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 0x7fb2880b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb28659c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 0x7fb2880b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 0x7fb2880b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:45722/0 (socket says 172.21.15.113:45722) 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb286d9d700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb2880b5050 0x7fb2880ae4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:37902/0 (socket says 172.21.15.113:37902) 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb28659c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 0x7fb2880b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:46910/0 (socket says 172.21.15.113:46910) 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 -- 172.21.15.113:0/724831442 learned_addr learned my addr 172.21.15.113:0/724831442 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 -- 172.21.15.113:0/724831442 >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 msgr2=0x7fb2880b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 --2- 172.21.15.113:0/724831442 >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 0x7fb2880b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 -- 172.21.15.113:0/724831442 >> v1:172.21.15.124:6789/0 conn(0x7fb2880b5050 legacy=0x7fb2880ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 -- 172.21.15.113:0/724831442 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288074d30 con 0x7fb288073f90 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb28659c700 1 --2- 172.21.15.113:0/724831442 >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 0x7fb2880b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:38.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb285d9b700 1 --2- 172.21.15.113:0/724831442 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 0x7fb2880b4b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb2780098f0 tx=0x7fb278009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16964036ea4b0db8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.639+0000 7fb28559a700 1 -- 172.21.15.113:0/724831442 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278011830 con 0x7fb288073f90 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28559a700 1 -- 172.21.15.113:0/724831442 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb278011990 con 0x7fb288073f90 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28559a700 1 -- 172.21.15.113:0/724831442 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278011d20 con 0x7fb288073f90 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 -- 172.21.15.113:0/724831442 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 msgr2=0x7fb2880b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 --2- 172.21.15.113:0/724831442 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 0x7fb2880b4b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb2780098f0 tx=0x7fb278009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 -- 172.21.15.113:0/724831442 shutdown_connections 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 --2- 172.21.15.113:0/724831442 >> v2:172.21.15.124:3300/0 conn(0x7fb28815e280 0x7fb2880b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 --2- 172.21.15.113:0/724831442 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb288073f90 0x7fb2880b4b10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 -- 172.21.15.113:0/724831442 >> 172.21.15.113:0/724831442 conn(0x7fb288072b20 msgr2=0x7fb28806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 -- 172.21.15.113:0/724831442 shutdown_connections 2024-01-16T13:52:38.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.640+0000 7fb28cf28700 1 -- 172.21.15.113:0/724831442 wait complete. 2024-01-16T13:52:38.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.641+0000 7fb28cf28700 1 Processor -- start 2024-01-16T13:52:38.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 -- start start 2024-01-16T13:52:38.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 0x7fb2881505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2880b5050 0x7fb2880b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb28815e6e0 0x7fb2880b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881f7930 con 0x7fb288073f90 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881f6cb0 con 0x7fb28815e6e0 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28cf28700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881f6f90 con 0x7fb2880b5050 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 0x7fb2881505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 0x7fb2881505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56068/0 (socket says 172.21.15.113:56068) 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.642+0000 7fb28659c700 1 -- 172.21.15.113:0/4204433679 learned_addr learned my addr 172.21.15.113:0/4204433679 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb285d9b700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2880b5050 0x7fb2880b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb286d9d700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb28815e6e0 0x7fb2880b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28659c700 1 -- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2880b5050 msgr2=0x7fb2880b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28659c700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2880b5050 0x7fb2880b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28659c700 1 -- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb28815e6e0 msgr2=0x7fb2880b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28659c700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb28815e6e0 0x7fb2880b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28659c700 1 -- 172.21.15.113:0/4204433679 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c0094a0 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb285d9b700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2880b5050 0x7fb2880b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb286d9d700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb28815e6e0 0x7fb2880b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28659c700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 0x7fb2881505f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb27c00ebe0 tx=0x7fb27c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c00c910 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c00ca70 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c00cde0 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb278009550 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.643+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2881f7550 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.644+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2881f76e0 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.644+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb27c013470 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.645+0000 7fb276ffd700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb26806b350 0x7fb26806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.645+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb27c089cb0 con 0x7fb288073f90 2024-01-16T13:52:38.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.645+0000 7fb285d9b700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb26806b350 0x7fb26806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.645+0000 7fb285d9b700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb26806b350 0x7fb26806d810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb2881589d0 tx=0x7fb278008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.650+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb27c054bb0 con 0x7fb288073f90 2024-01-16T13:52:38.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.839+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2881f7ba0 con 0x7fb288073f90 2024-01-16T13:52:38.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.839+0000 7fb276ffd700 1 -- 172.21.15.113:0/4204433679 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb27c0170a0 con 0x7fb288073f90 2024-01-16T13:52:38.822 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.841+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb26806b350 msgr2=0x7fb26806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.841+0000 7fb28cf28700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb26806b350 0x7fb26806d810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb2881589d0 tx=0x7fb278008030 comp rx=0 tx=0).stop 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.841+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 msgr2=0x7fb2881505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.841+0000 7fb28cf28700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 0x7fb2881505f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb27c00ebe0 tx=0x7fb27c009ec0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 shutdown_connections 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fb26806b350 0x7fb26806d810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb28815e6e0 0x7fb2880b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb288073f90 0x7fb2881505f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 --2- 172.21.15.113:0/4204433679 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb2880b5050 0x7fb2880b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 >> 172.21.15.113:0/4204433679 conn(0x7fb288072b20 msgr2=0x7fb28815d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:38.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 shutdown_connections 2024-01-16T13:52:38.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.842+0000 7fb28cf28700 1 -- 172.21.15.113:0/4204433679 wait complete. 2024-01-16T13:52:38.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:38.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.977+0000 7fcee39ca700 1 Processor -- start 2024-01-16T13:52:38.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.977+0000 7fcee39ca700 1 -- start start 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee39ca700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc158570 0x7fcedc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee39ca700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc158950 con 0x7fcedc074710 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee39ca700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc15e670 con 0x7fcedc06e860 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee39ca700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc15f2f0 con 0x7fcedc158570 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee1f67700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fcedc074710 0x7fcedc06b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38760/0 (socket says 172.21.15.113:38760) 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee1f67700 1 -- 172.21.15.113:0/954425740 learned_addr learned my addr 172.21.15.113:0/954425740 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fcee1766700 1 --2- 172.21.15.113:0/954425740 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc158570 0x7fcedc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620881918 0 0) 0x7fcedc158950 con 0x7fcedc074710 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.978+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec80035b0 con 0x7fcedc074710 2024-01-16T13:52:38.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752907188 0 0) 0x7fcedc15e670 con 0x7fcedc06e860 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcedc158950 con 0x7fcedc06e860 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fcee1766700 1 -- 172.21.15.113:0/954425740 >> v1:172.21.15.124:6789/0 conn(0x7fcedc06e860 legacy=0x7fcedc074000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fcee1766700 1 -- 172.21.15.113:0/954425740 >> v1:172.21.15.113:6789/0 conn(0x7fcedc074710 legacy=0x7fcedc06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459325539 0 0) 0x7fcec80035b0 con 0x7fcedc074710 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fcee1766700 1 -- 172.21.15.113:0/954425740 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc15e950 con 0x7fcedc158570 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fcee1766700 1 --2- 172.21.15.113:0/954425740 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc158570 0x7fcedc06e320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009870 tx=0x7fcecc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fca47e25344b23 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced40019f0 con 0x7fcedc158570 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc004030 con 0x7fcedc158570 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.979+0000 7fced3fff700 1 -- 172.21.15.113:0/954425740 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0043a0 con 0x7fcedc158570 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 -- 172.21.15.113:0/954425740 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc158570 msgr2=0x7fcedc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 --2- 172.21.15.113:0/954425740 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc158570 0x7fcedc06e320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009870 tx=0x7fcecc009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 -- 172.21.15.113:0/954425740 shutdown_connections 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 --2- 172.21.15.113:0/954425740 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc158570 0x7fcedc06e320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 -- 172.21.15.113:0/954425740 >> 172.21.15.113:0/954425740 conn(0x7fcedc0cd6b0 msgr2=0x7fcedc0cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:38.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 -- 172.21.15.113:0/954425740 shutdown_connections 2024-01-16T13:52:38.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.980+0000 7fcee39ca700 1 -- 172.21.15.113:0/954425740 wait complete. 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.981+0000 7fcee39ca700 1 Processor -- start 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.981+0000 7fcee39ca700 1 -- start start 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee39ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 0x7fcedc149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee39ca700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc074710 0x7fcedc0add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee39ca700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcedc158570 0x7fcedc0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee39ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0ae8b0 con 0x7fcedc06e860 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee39ca700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0aeb90 con 0x7fcedc158570 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee39ca700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0aee70 con 0x7fcedc074710 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee1766700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 0x7fcedc149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee1766700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 0x7fcedc149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56076/0 (socket says 172.21.15.113:56076) 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.982+0000 7fcee1766700 1 -- 172.21.15.113:0/1110500409 learned_addr learned my addr 172.21.15.113:0/1110500409 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee0f65700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc074710 0x7fcedc0add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1f67700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcedc158570 0x7fcedc0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1766700 1 -- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc074710 msgr2=0x7fcedc0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1766700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc074710 0x7fcedc0add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1766700 1 -- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcedc158570 msgr2=0x7fcedc0ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1766700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcedc158570 0x7fcedc0ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1766700 1 -- 172.21.15.113:0/1110500409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc0094a0 con 0x7fcedc06e860 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1f67700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcedc158570 0x7fcedc0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee1766700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 0x7fcedc149460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0090b0 tx=0x7fcecc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc003c30 con 0x7fcedc06e860 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc01f070 con 0x7fcedc06e860 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc011630 con 0x7fcedc06e860 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc0af150 con 0x7fcedc06e860 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.983+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc1537e0 con 0x7fcedc06e860 2024-01-16T13:52:38.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.984+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc06e190 con 0x7fcedc06e860 2024-01-16T13:52:38.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.984+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcecc021030 con 0x7fcedc06e860 2024-01-16T13:52:38.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.985+0000 7fced1ffb700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcec006b350 0x7fcec006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:38.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.985+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcecc08ae90 con 0x7fcedc06e860 2024-01-16T13:52:38.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.985+0000 7fcee0f65700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcec006b350 0x7fcec006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:38.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.985+0000 7fcee0f65700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcec006b350 0x7fcec006d810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcedc15e070 tx=0x7fced8009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:38.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:38.989+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcecc055d90 con 0x7fcedc06e860 2024-01-16T13:52:39.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.178+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcedc069020 con 0x7fcedc06e860 2024-01-16T13:52:39.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.179+0000 7fced1ffb700 1 -- 172.21.15.113:0/1110500409 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcecc0593b0 con 0x7fcedc06e860 2024-01-16T13:52:39.162 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:39.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcec006b350 msgr2=0x7fcec006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcec006b350 0x7fcec006d810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcedc15e070 tx=0x7fced8009040 comp rx=0 tx=0).stop 2024-01-16T13:52:39.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 msgr2=0x7fcedc149460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 0x7fcedc149460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0090b0 tx=0x7fcecc004750 comp rx=0 tx=0).stop 2024-01-16T13:52:39.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 shutdown_connections 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fcec006b350 0x7fcec006d810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcedc158570 0x7fcedc0ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcedc06e860 0x7fcedc149460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 --2- 172.21.15.113:0/1110500409 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcedc074710 0x7fcedc0add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 >> 172.21.15.113:0/1110500409 conn(0x7fcedc0cd6b0 msgr2=0x7fcedc06bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 shutdown_connections 2024-01-16T13:52:39.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.181+0000 7fcee39ca700 1 -- 172.21.15.113:0/1110500409 wait complete. 2024-01-16T13:52:39.176 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:39.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.317+0000 7fda45582700 1 Processor -- start 2024-01-16T13:52:39.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.317+0000 7fda45582700 1 -- start start 2024-01-16T13:52:39.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda45582700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda40161510 0x7fda400cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda45582700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fda400b98e0 0x7fda400cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda45582700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda40162af0 con 0x7fda400b59d0 2024-01-16T13:52:39.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda45582700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda400d13e0 con 0x7fda40161510 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda45582700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda40161f70 con 0x7fda400b98e0 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda40161510 0x7fda400cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3f7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fda400b59d0 0x7fda400b5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38764/0 (socket says 172.21.15.113:38764) 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3effd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda40161510 0x7fda400cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56092/0 (socket says 172.21.15.113:56092) 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3f7fe700 1 -- 172.21.15.113:0/1080283674 learned_addr learned my addr 172.21.15.113:0/1080283674 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3e7fc700 1 --2- 172.21.15.113:0/1080283674 >> v2:172.21.15.124:3300/0 conn(0x7fda400b98e0 0x7fda400cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3effd700 1 -- 172.21.15.113:0/1080283674 >> v2:172.21.15.124:3300/0 conn(0x7fda400b98e0 msgr2=0x7fda400cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3effd700 1 --2- 172.21.15.113:0/1080283674 >> v2:172.21.15.124:3300/0 conn(0x7fda400b98e0 0x7fda400cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3effd700 1 -- 172.21.15.113:0/1080283674 >> v1:172.21.15.113:6789/0 conn(0x7fda400b59d0 legacy=0x7fda400b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3effd700 1 -- 172.21.15.113:0/1080283674 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda40162250 con 0x7fda40161510 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.318+0000 7fda3dffb700 1 -- 172.21.15.113:0/1080283674 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487348246 0 0) 0x7fda40162af0 con 0x7fda400b59d0 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda3effd700 1 --2- 172.21.15.113:0/1080283674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cec20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda400ce2c0 tx=0x7fda34009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb932a6a8752be6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda3dffb700 1 -- 172.21.15.113:0/1080283674 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda34011830 con 0x7fda40161510 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda3dffb700 1 -- 172.21.15.113:0/1080283674 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda34011990 con 0x7fda40161510 2024-01-16T13:52:39.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda3dffb700 1 -- 172.21.15.113:0/1080283674 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda34011d20 con 0x7fda40161510 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 -- 172.21.15.113:0/1080283674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 msgr2=0x7fda400cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 --2- 172.21.15.113:0/1080283674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cec20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda400ce2c0 tx=0x7fda34009b20 comp rx=0 tx=0).stop 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 -- 172.21.15.113:0/1080283674 shutdown_connections 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 --2- 172.21.15.113:0/1080283674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cec20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 --2- 172.21.15.113:0/1080283674 >> v2:172.21.15.124:3300/0 conn(0x7fda400b98e0 0x7fda400cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 -- 172.21.15.113:0/1080283674 >> 172.21.15.113:0/1080283674 conn(0x7fda4006b410 msgr2=0x7fda4006b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 -- 172.21.15.113:0/1080283674 shutdown_connections 2024-01-16T13:52:39.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.319+0000 7fda45582700 1 -- 172.21.15.113:0/1080283674 wait complete. 2024-01-16T13:52:39.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 Processor -- start 2024-01-16T13:52:39.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 -- start start 2024-01-16T13:52:39.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda400b59d0 0x7fda400d2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda400b98e0 0x7fda400d2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda400cce30 con 0x7fda40161510 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda400cbf50 con 0x7fda400b59d0 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda45582700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda400cbc70 con 0x7fda400b98e0 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.321+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56108/0 (socket says 172.21.15.113:56108) 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 -- 172.21.15.113:0/2192950843 learned_addr learned my addr 172.21.15.113:0/2192950843 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3effd700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda400b59d0 0x7fda400d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3e7fc700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda400b98e0 0x7fda400d2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 -- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda400b98e0 msgr2=0x7fda400d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda400b98e0 0x7fda400d2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 -- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda400b59d0 msgr2=0x7fda400d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda400b59d0 0x7fda400d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 -- 172.21.15.113:0/2192950843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340094a0 con 0x7fda40161510 2024-01-16T13:52:39.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3effd700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda400b59d0 0x7fda400d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda3f7fe700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cb020 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fda38002230 tx=0x7fda3800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3800ab70 con 0x7fda40161510 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3800acf0 con 0x7fda40161510 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda400c7830 con 0x7fda40161510 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38007310 con 0x7fda40161510 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.322+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda400c7df0 con 0x7fda40161510 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.323+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda40068450 con 0x7fda40161510 2024-01-16T13:52:39.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.324+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fda38007470 con 0x7fda40161510 2024-01-16T13:52:39.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.324+0000 7fda277fe700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda2c06b350 0x7fda2c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.324+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda38089a40 con 0x7fda40161510 2024-01-16T13:52:39.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.324+0000 7fda3effd700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda2c06b350 0x7fda2c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.324+0000 7fda3effd700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda2c06b350 0x7fda2c06d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fda340090b0 tx=0x7fda3401c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:39.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.328+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda3808be60 con 0x7fda40161510 2024-01-16T13:52:39.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.518+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda400c8010 con 0x7fda40161510 2024-01-16T13:52:39.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.518+0000 7fda277fe700 1 -- 172.21.15.113:0/2192950843 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fda38018020 con 0x7fda40161510 2024-01-16T13:52:39.501 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda2c06b350 msgr2=0x7fda2c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda2c06b350 0x7fda2c06d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fda340090b0 tx=0x7fda3401c040 comp rx=0 tx=0).stop 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 msgr2=0x7fda400cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cb020 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fda38002230 tx=0x7fda3800cf50 comp rx=0 tx=0).stop 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 shutdown_connections 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda2c06b350 0x7fda2c06d810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda40161510 0x7fda400cb020 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda400b59d0 0x7fda400d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 --2- 172.21.15.113:0/2192950843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda400b98e0 0x7fda400d2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 >> 172.21.15.113:0/2192950843 conn(0x7fda4006b410 msgr2=0x7fda400b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 shutdown_connections 2024-01-16T13:52:39.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.520+0000 7fda45582700 1 -- 172.21.15.113:0/2192950843 wait complete. 2024-01-16T13:52:39.515 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T13:52:39.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.654+0000 7f3706385700 1 Processor -- start 2024-01-16T13:52:39.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.655+0000 7f3706385700 1 -- start start 2024-01-16T13:52:39.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.655+0000 7f3706385700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f370015ac50 0x7f370006c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.655+0000 7f3706385700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 0x7f37000b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.655+0000 7f3706385700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370006bf50 con 0x7f37000b4ef0 2024-01-16T13:52:39.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.655+0000 7f3706385700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37000b5380 con 0x7f370015ac50 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.655+0000 7f3706385700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370015b030 con 0x7f370006f460 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36ff7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 0x7f37000b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f370015ac50 0x7f370006c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36ff7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 0x7f37000b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:45768/0 (socket says 172.21.15.113:45768) 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f370015ac50 0x7f370006c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:46948/0 (socket says 172.21.15.113:46948) 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36ff7fe700 1 -- 172.21.15.113:0/3256682595 learned_addr learned my addr 172.21.15.113:0/3256682595 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36feffd700 1 -- 172.21.15.113:0/3256682595 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726555934 0 0) 0x7f370006bf50 con 0x7f37000b4ef0 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36feffd700 1 -- 172.21.15.113:0/3256682595 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec0035b0 con 0x7f37000b4ef0 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 -- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 msgr2=0x7f37000b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 --2- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 0x7f37000b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 -- 172.21.15.113:0/3256682595 >> v1:172.21.15.124:6789/0 conn(0x7f37000b4ef0 legacy=0x7f37000cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 -- 172.21.15.113:0/3256682595 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370006e810 con 0x7f370015ac50 2024-01-16T13:52:39.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36ff7fe700 1 --2- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 0x7f37000b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.656+0000 7f36fffff700 1 --2- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f370015ac50 0x7f370006c3e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009840 tx=0x7f36f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bdfbad5cb494b2b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f36feffd700 1 -- 172.21.15.113:0/3256682595 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0011830 con 0x7f370015ac50 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f36feffd700 1 -- 172.21.15.113:0/3256682595 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0011990 con 0x7f370015ac50 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f36feffd700 1 -- 172.21.15.113:0/3256682595 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0011d20 con 0x7f370015ac50 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 -- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f370015ac50 msgr2=0x7f370006c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 --2- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f370015ac50 0x7f370006c3e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009840 tx=0x7f36f0009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 -- 172.21.15.113:0/3256682595 shutdown_connections 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 --2- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f370015ac50 0x7f370006c3e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 --2- 172.21.15.113:0/3256682595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370006f460 0x7f37000b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 -- 172.21.15.113:0/3256682595 >> 172.21.15.113:0/3256682595 conn(0x7f37000b3130 msgr2=0x7f37000d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 -- 172.21.15.113:0/3256682595 shutdown_connections 2024-01-16T13:52:39.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.657+0000 7f3706385700 1 -- 172.21.15.113:0/3256682595 wait complete. 2024-01-16T13:52:39.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.658+0000 7f3706385700 1 Processor -- start 2024-01-16T13:52:39.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 -- start start 2024-01-16T13:52:39.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 0x7f370014bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37000b4ef0 0x7f37001527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370015ac50 0x7f3700158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370015b380 con 0x7f370006f460 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3700152d20 con 0x7f37000b4ef0 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f3706385700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37001516d0 con 0x7f370015ac50 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.659+0000 7f36fffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 0x7f370014bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 0x7f370014bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56122/0 (socket says 172.21.15.113:56122) 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 -- 172.21.15.113:0/3086125843 learned_addr learned my addr 172.21.15.113:0/3086125843 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36ff7fe700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37000b4ef0 0x7f37001527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f3704922700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370015ac50 0x7f3700158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 -- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370015ac50 msgr2=0x7f3700158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370015ac50 0x7f3700158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 -- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37000b4ef0 msgr2=0x7f37001527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37000b4ef0 0x7f37001527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 -- 172.21.15.113:0/3086125843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e800b7f0 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fffff700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 0x7f370014bc60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f36f00090b0 tx=0x7f36f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f00036a0 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f001e070 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f00094a0 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0003a40 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.660+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3700151c90 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.661+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370015b170 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.662+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f36f0003ba0 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.662+0000 7f36fcff9700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f36e006b350 0x7f36e006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.662+0000 7f36ff7fe700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f36e006b350 0x7f36e006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.662+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36f008aa00 con 0x7f370006f460 2024-01-16T13:52:39.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.663+0000 7f36ff7fe700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f36e006b350 0x7f36e006d810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37000b4d10 tx=0x7f36e800b4d0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:39.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.667+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36f0055880 con 0x7f370006f460 2024-01-16T13:52:39.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.872+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3700151e80 con 0x7f370006f460 2024-01-16T13:52:39.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.873+0000 7f36fcff9700 1 -- 172.21.15.113:0/3086125843 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f36f0020080 con 0x7f370006f460 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f36e006b350 msgr2=0x7f36e006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f36e006b350 0x7f36e006d810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37000b4d10 tx=0x7f36e800b4d0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 msgr2=0x7f370014bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 0x7f370014bc60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f36f00090b0 tx=0x7f36f0004750 comp rx=0 tx=0).stop 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 shutdown_connections 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f36e006b350 0x7f36e006d810 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f370015ac50 0x7f3700158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f370006f460 0x7f370014bc60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 --2- 172.21.15.113:0/3086125843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37000b4ef0 0x7f37001527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:39.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 >> 172.21.15.113:0/3086125843 conn(0x7f37000b3130 msgr2=0x7f3700159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:39.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 shutdown_connections 2024-01-16T13:52:39.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:39.875+0000 7f3706385700 1 -- 172.21.15.113:0/3086125843 wait complete. 2024-01-16T13:52:39.859 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-16T13:52:39.870 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-01-16T13:52:39.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:39.870 DEBUG:teuthology.orchestra.run.smithi124:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-01-16T13:52:39.911 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:52:39.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:39.911 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T13:52:39.990 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:39.990 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:40.008 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:40.008 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:40.066 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:52:40.066 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:40.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:40.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.301+0000 7f320fe69700 1 Processor -- start 2024-01-16T13:52:40.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.302+0000 7f320fe69700 1 -- start start 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.302+0000 7f320fe69700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f320815a3b0 0x7f32080b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.302+0000 7f320fe69700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320815a790 con 0x7f32080cf360 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.302+0000 7f320fe69700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32080b4c40 con 0x7f320815deb0 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.302+0000 7f320fe69700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32080cf8b0 con 0x7f320815a3b0 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f320815a3b0 0x7f32080b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f320815a3b0 0x7f32080b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56138/0 (socket says 172.21.15.113:56138) 2024-01-16T13:52:40.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 -- 172.21.15.113:0/1689353410 learned_addr learned my addr 172.21.15.113:0/1689353410 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 -- 172.21.15.113:0/1689353410 >> v1:172.21.15.124:6789/0 conn(0x7f320815deb0 legacy=0x7f320815e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 -- 172.21.15.113:0/1689353410 >> v1:172.21.15.113:6789/0 conn(0x7f32080cf360 legacy=0x7f32080b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 -- 172.21.15.113:0/1689353410 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3208070c60 con 0x7f320815a3b0 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320cc03700 1 -- 172.21.15.113:0/1689353410 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634212209 0 0) 0x7f320815a790 con 0x7f32080cf360 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320dc05700 1 --2- 172.21.15.113:0/1689353410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32080b3320 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00bb90 tx=0x7f31fc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f1b20b0dfb7138 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320cc03700 1 -- 172.21.15.113:0/1689353410 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00dd50 con 0x7f320815a3b0 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320cc03700 1 -- 172.21.15.113:0/1689353410 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc010470 con 0x7f320815a3b0 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.303+0000 7f320cc03700 1 -- 172.21.15.113:0/1689353410 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc010800 con 0x7f320815a3b0 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 -- 172.21.15.113:0/1689353410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 msgr2=0x7f32080b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 --2- 172.21.15.113:0/1689353410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32080b3320 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00bb90 tx=0x7f31fc00bea0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 -- 172.21.15.113:0/1689353410 shutdown_connections 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 --2- 172.21.15.113:0/1689353410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32080b3320 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 -- 172.21.15.113:0/1689353410 >> 172.21.15.113:0/1689353410 conn(0x7f32080b1740 msgr2=0x7f32080b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:40.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 -- 172.21.15.113:0/1689353410 shutdown_connections 2024-01-16T13:52:40.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.304+0000 7f320fe69700 1 -- 172.21.15.113:0/1689353410 wait complete. 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.305+0000 7f320fe69700 1 Processor -- start 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.305+0000 7f320fe69700 1 -- start start 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320fe69700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32080cf360 0x7f3208140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320fe69700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32081567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320fe69700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f320815deb0 0x7f3208156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320fe69700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320815d2f0 con 0x7f320815a3b0 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320fe69700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320815d540 con 0x7f32080cf360 2024-01-16T13:52:40.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320fe69700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320815d820 con 0x7f320815deb0 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32081567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32081567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56154/0 (socket says 172.21.15.113:56154) 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 -- 172.21.15.113:0/709202692 learned_addr learned my addr 172.21.15.113:0/709202692 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320dc05700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32080cf360 0x7f3208140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320e406700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f320815deb0 0x7f3208156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 -- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f320815deb0 msgr2=0x7f3208156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f320815deb0 0x7f3208156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 -- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32080cf360 msgr2=0x7f3208140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32080cf360 0x7f3208140460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320d404700 1 -- 172.21.15.113:0/709202692 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc00b7f0 con 0x7f320815a3b0 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320dc05700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32080cf360 0x7f3208140460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.306+0000 7f320e406700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f320815deb0 0x7f3208156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:40.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.307+0000 7f320d404700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32081567e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3204001eb0 tx=0x7f320400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.307+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040054e0 con 0x7f320815a3b0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.307+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3204003680 con 0x7f320815a3b0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.307+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204003a30 con 0x7f320815a3b0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.307+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32081e4b00 con 0x7f320815a3b0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.307+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32081e50c0 con 0x7f320815a3b0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.308+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32080751e0 con 0x7f320815a3b0 2024-01-16T13:52:40.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.308+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3204003b90 con 0x7f320815a3b0 2024-01-16T13:52:40.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.309+0000 7f31fa7fc700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f31ec06b350 0x7f31ec06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.309+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3204088f30 con 0x7f320815a3b0 2024-01-16T13:52:40.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.309+0000 7f320dc05700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f31ec06b350 0x7f31ec06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.309+0000 7f320dc05700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f31ec06b350 0x7f31ec06d810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00bee0 tx=0x7f31fc013000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:40.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.313+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3204053e30 con 0x7f320815a3b0 2024-01-16T13:52:40.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:52:40.469 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:52:40.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.507+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32080b38d0 con 0x7f320815a3b0 2024-01-16T13:52:40.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.508+0000 7f31fa7fc700 1 -- 172.21.15.113:0/709202692 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f3204017090 con 0x7f320815a3b0 2024-01-16T13:52:40.491 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:40.491 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f31ec06b350 msgr2=0x7f31ec06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f31ec06b350 0x7f31ec06d810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00bee0 tx=0x7f31fc013000 comp rx=0 tx=0).stop 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 msgr2=0x7f32081567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32081567e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3204001eb0 tx=0x7f320400c7e0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 shutdown_connections 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f31ec06b350 0x7f31ec06d810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f320815deb0 0x7f3208156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32080cf360 0x7f3208140460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 --2- 172.21.15.113:0/709202692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f320815a3b0 0x7f32081567e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 >> 172.21.15.113:0/709202692 conn(0x7f32080b1740 msgr2=0x7f32080ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 shutdown_connections 2024-01-16T13:52:40.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.510+0000 7f320fe69700 1 -- 172.21.15.113:0/709202692 wait complete. 2024-01-16T13:52:40.506 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:40.506 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:40.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.700+0000 7f3106b46700 1 Processor -- start 2024-01-16T13:52:40.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.700+0000 7f3106b46700 1 -- start start 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f3106b46700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f3106b46700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31000d03a0 0x7f310015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f3106b46700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31000b3860 con 0x7f310015f700 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f3106b46700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310015fb90 con 0x7f31000d03a0 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f3106b46700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31000b2520 con 0x7f31000cffc0 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f31050e3700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f310015f700 0x7f310009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38776/0 (socket says 172.21.15.113:38776) 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f30fffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f31000d03a0 0x7f310015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.701+0000 7f31050e3700 1 -- 172.21.15.113:0/971834979 learned_addr learned my addr 172.21.15.113:0/971834979 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f31048e2700 1 --2- 172.21.15.113:0/971834979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30ff7fe700 1 -- 172.21.15.113:0/971834979 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277749451 0 0) 0x7f30f8001490 con 0x7f310015f700 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30ff7fe700 1 -- 172.21.15.113:0/971834979 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f00035b0 con 0x7f310015f700 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30fffff700 1 -- 172.21.15.113:0/971834979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 msgr2=0x7f31000b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30fffff700 1 --2- 172.21.15.113:0/971834979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30fffff700 1 -- 172.21.15.113:0/971834979 >> v1:172.21.15.113:6789/0 conn(0x7f310015f700 legacy=0x7f310009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30fffff700 1 -- 172.21.15.113:0/971834979 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000b2800 con 0x7f31000d03a0 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30fffff700 1 --2- 172.21.15.113:0/971834979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31000d03a0 0x7f310015f1c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009840 tx=0x7f30ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14ba7f0f6d2a84f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30ff7fe700 1 -- 172.21.15.113:0/971834979 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec011830 con 0x7f31000d03a0 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30ff7fe700 1 -- 172.21.15.113:0/971834979 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec011990 con 0x7f31000d03a0 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.702+0000 7f30ff7fe700 1 -- 172.21.15.113:0/971834979 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec011d20 con 0x7f31000d03a0 2024-01-16T13:52:40.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 -- 172.21.15.113:0/971834979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31000d03a0 msgr2=0x7f310015f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 --2- 172.21.15.113:0/971834979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31000d03a0 0x7f310015f1c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009840 tx=0x7f30ec009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 -- 172.21.15.113:0/971834979 shutdown_connections 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 --2- 172.21.15.113:0/971834979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 --2- 172.21.15.113:0/971834979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31000d03a0 0x7f310015f1c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 -- 172.21.15.113:0/971834979 >> 172.21.15.113:0/971834979 conn(0x7f31000d1e50 msgr2=0x7f31000d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 -- 172.21.15.113:0/971834979 shutdown_connections 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.703+0000 7f3106b46700 1 -- 172.21.15.113:0/971834979 wait complete. 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.704+0000 7f3106b46700 1 Processor -- start 2024-01-16T13:52:40.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.704+0000 7f3106b46700 1 -- start start 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f3106b46700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f3106b46700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31000d03a0 0x7f31001e5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f3106b46700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 0x7f31001e5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f3106b46700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31001f5d60 con 0x7f310015f700 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f3106b46700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31001f50e0 con 0x7f31000d03a0 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f3106b46700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31001f53c0 con 0x7f31000cffc0 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f31050e3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 0x7f31001e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f31050e3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 0x7f31001e5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56170/0 (socket says 172.21.15.113:56170) 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.705+0000 7f31050e3700 1 -- 172.21.15.113:0/1395159450 learned_addr learned my addr 172.21.15.113:0/1395159450 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31050e3700 1 -- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 msgr2=0x7f31000b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31048e2700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f30fffff700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31000d03a0 0x7f31001e5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31050e3700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31050e3700 1 -- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31000d03a0 msgr2=0x7f31001e5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31050e3700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31000d03a0 0x7f31001e5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31050e3700 1 -- 172.21.15.113:0/1395159450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec0094a0 con 0x7f310015f700 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31048e2700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f30fffff700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31000d03a0 0x7f31001e5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f31050e3700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 0x7f31001e5990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f30f800cb40 tx=0x7f30f800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800ab50 con 0x7f310015f700 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f800acb0 con 0x7f310015f700 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31001f56a0 con 0x7f310015f700 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8012620 con 0x7f310015f700 2024-01-16T13:52:40.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.706+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31000b4d10 con 0x7f310015f700 2024-01-16T13:52:40.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.707+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31001f5ea0 con 0x7f310015f700 2024-01-16T13:52:40.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.708+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f30f801b3b0 con 0x7f310015f700 2024-01-16T13:52:40.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.708+0000 7f30fd7fa700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f30e406b350 0x7f30e406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:40.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.708+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30f80897a0 con 0x7f310015f700 2024-01-16T13:52:40.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.708+0000 7f31048e2700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f30e406b350 0x7f30e406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:40.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.709+0000 7f31048e2700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f30e406b350 0x7f30e406d810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009ac0 tx=0x7f30f4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:40.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.713+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31001f5ea0 con 0x7f310015f700 2024-01-16T13:52:40.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.902+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31000b3860 con 0x7f310015f700 2024-01-16T13:52:40.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.902+0000 7f30fd7fa700 1 -- 172.21.15.113:0/1395159450 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30f8017020 con 0x7f310015f700 2024-01-16T13:52:40.886 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.904+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f30e406b350 msgr2=0x7f30e406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.904+0000 7f3106b46700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f30e406b350 0x7f30e406d810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009ac0 tx=0x7f30f4008040 comp rx=0 tx=0).stop 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.904+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 msgr2=0x7f31001e5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.904+0000 7f3106b46700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 0x7f31001e5990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f30f800cb40 tx=0x7f30f800cf00 comp rx=0 tx=0).stop 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 shutdown_connections 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f30e406b350 0x7f30e406d810 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f310015f700 0x7f31001e5990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31000cffc0 0x7f31000b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 --2- 172.21.15.113:0/1395159450 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31000d03a0 0x7f31001e5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 >> 172.21.15.113:0/1395159450 conn(0x7f31000d1e50 msgr2=0x7f310015d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 shutdown_connections 2024-01-16T13:52:40.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:40.905+0000 7f3106b46700 1 -- 172.21.15.113:0/1395159450 wait complete. 2024-01-16T13:52:40.899 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:41.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.040+0000 7fc255141700 1 Processor -- start 2024-01-16T13:52:41.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.040+0000 7fc255141700 1 -- start start 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc255141700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc25015e280 0x7fc2500b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc255141700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc2500b5220 0x7fc25006e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc255141700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2500b42a0 con 0x7fc250073f90 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc255141700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2500af900 con 0x7fc2500b5220 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc255141700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc250074d30 con 0x7fc25015e280 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc24e59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc250073f90 0x7fc2500b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38790/0 (socket says 172.21.15.113:38790) 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.041+0000 7fc24e59c700 1 -- 172.21.15.113:0/3999814255 learned_addr learned my addr 172.21.15.113:0/3999814255 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc25015e280 0x7fc2500b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24f59e700 1 --2- 172.21.15.113:0/3999814255 >> v2:172.21.15.124:3300/0 conn(0x7fc2500b5220 0x7fc25006e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532476603 0 0) 0x7fc240001110 con 0x7fc250073f90 2024-01-16T13:52:41.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc23c0035b0 con 0x7fc250073f90 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390598799 0 0) 0x7fc23c0035b0 con 0x7fc250073f90 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc25015e280 msgr2=0x7fc2500b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc25015e280 0x7fc2500b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 >> v2:172.21.15.124:3300/0 conn(0x7fc2500b5220 msgr2=0x7fc25006e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 --2- 172.21.15.113:0/3999814255 >> v2:172.21.15.124:3300/0 conn(0x7fc2500b5220 0x7fc25006e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc250074490 con 0x7fc250073f90 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fc240001800 con 0x7fc250073f90 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 >> v1:172.21.15.113:6789/0 conn(0x7fc250073f90 legacy=0x7fc2500b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc23c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc23c004850 0x7fc23c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0051d0 0x7fc23c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500af900 con 0x7fc23c003a00 2024-01-16T13:52:41.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc250074d30 con 0x7fc23c0051d0 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c004650 con 0x7fc23c004850 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc23c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24f59e700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0051d0 0x7fc23c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24e59c700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc23c004850 0x7fc23c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 -- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc23c004850 msgr2=0x7fc23c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc23c004850 0x7fc23c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 -- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0051d0 msgr2=0x7fc23c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0051d0 0x7fc23c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 -- 172.21.15.113:0/3999814255 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c0035b0 con 0x7fc23c003a00 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24f59e700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0051d0 0x7fc23c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.042+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc23c003e20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc238009b60 tx=0x7fc23c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483991f8d657ac03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23800e6c0 con 0x7fc23c003a00 2024-01-16T13:52:41.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23800e820 con 0x7fc23c003a00 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc24dd9b700 1 -- 172.21.15.113:0/3999814255 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23800eb90 con 0x7fc23c003a00 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc255141700 1 -- 172.21.15.113:0/3999814255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 msgr2=0x7fc23c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc255141700 1 --2- 172.21.15.113:0/3999814255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc23c003e20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc238009b60 tx=0x7fc23c005880 comp rx=0 tx=0).stop 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc24f59e700 1 -- 172.21.15.113:0/3999814255 reap_dead start 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc255141700 1 -- 172.21.15.113:0/3999814255 shutdown_connections 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc255141700 1 -- 172.21.15.113:0/3999814255 >> 172.21.15.113:0/3999814255 conn(0x7fc250072b20 msgr2=0x7fc25006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc255141700 1 -- 172.21.15.113:0/3999814255 shutdown_connections 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.043+0000 7fc255141700 1 -- 172.21.15.113:0/3999814255 wait complete. 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 Processor -- start 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 -- start start 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc25014a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc250161480 0x7fc2500c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2500c7c20 0x7fc2500c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500c85e0 con 0x7fc23c003a00 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500c7580 con 0x7fc250161480 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc255141700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500c6fc0 con 0x7fc2500c7c20 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.045+0000 7fc24ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc25014a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc25014a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56182/0 (socket says 172.21.15.113:56182) 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 -- 172.21.15.113:0/2576684139 learned_addr learned my addr 172.21.15.113:0/2576684139 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24e59c700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc250161480 0x7fc2500c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24f59e700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2500c7c20 0x7fc2500c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 -- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2500c7c20 msgr2=0x7fc2500c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2500c7c20 0x7fc2500c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 -- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc250161480 msgr2=0x7fc2500c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc250161480 0x7fc2500c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 -- 172.21.15.113:0/2576684139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500c72a0 con 0x7fc23c003a00 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24e59c700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc250161480 0x7fc2500c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24f59e700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2500c7c20 0x7fc2500c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc24ed9d700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc25014a520 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0045a0 tx=0x7fc2380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23800e6c0 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc238003ab0 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25015e810 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238003e60 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.046+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25014aa60 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.047+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc250069020 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.048+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc238013470 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.048+0000 7fc2477fe700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc23006b350 0x7fc23006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.048+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc23808b520 con 0x7fc23c003a00 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.048+0000 7fc24e59c700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc23006b350 0x7fc23006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.049+0000 7fc24e59c700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc23006b350 0x7fc23006d810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc2400012b0 tx=0x7fc240009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.053+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc238056540 con 0x7fc23c003a00 2024-01-16T13:52:41.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.240+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc25015ee60 con 0x7fc23c003a00 2024-01-16T13:52:41.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.241+0000 7fc2477fe700 1 -- 172.21.15.113:0/2576684139 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc238059b60 con 0x7fc23c003a00 2024-01-16T13:52:41.224 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc23006b350 msgr2=0x7fc23006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc23006b350 0x7fc23006d810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc2400012b0 tx=0x7fc240009040 comp rx=0 tx=0).stop 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 msgr2=0x7fc25014a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc25014a520 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0045a0 tx=0x7fc2380047c0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 shutdown_connections 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc23006b350 0x7fc23006d810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2500c7c20 0x7fc2500c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc23c003a00 0x7fc25014a520 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 --2- 172.21.15.113:0/2576684139 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc250161480 0x7fc2500c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 >> 172.21.15.113:0/2576684139 conn(0x7fc250072b20 msgr2=0x7fc2500b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:41.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 shutdown_connections 2024-01-16T13:52:41.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.243+0000 7fc255141700 1 -- 172.21.15.113:0/2576684139 wait complete. 2024-01-16T13:52:41.238 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:41.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.378+0000 7effb4851700 1 Processor -- start 2024-01-16T13:52:41.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.378+0000 7effb4851700 1 -- start start 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.379+0000 7effb4851700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effb006c2e0 0x7effb015b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.379+0000 7effb4851700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb006eeb0 con 0x7effb015b240 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.379+0000 7effb4851700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb006f190 con 0x7effb015bb60 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.379+0000 7effb4851700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb006f470 con 0x7effb006c2e0 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effadd9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effb006c2e0 0x7effb015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effae59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7effb015b240 0x7effb006e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38792/0 (socket says 172.21.15.113:38792) 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effae59c700 1 -- 172.21.15.113:0/1564264383 learned_addr learned my addr 172.21.15.113:0/1564264383 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effad59a700 1 -- 172.21.15.113:0/1564264383 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881121615 0 0) 0x7effb006eeb0 con 0x7effb015b240 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effad59a700 1 -- 172.21.15.113:0/1564264383 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff9c0035b0 con 0x7effb015b240 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effadd9b700 1 -- 172.21.15.113:0/1564264383 >> v1:172.21.15.124:6789/0 conn(0x7effb015bb60 legacy=0x7effb01647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effadd9b700 1 -- 172.21.15.113:0/1564264383 >> v1:172.21.15.113:6789/0 conn(0x7effb015b240 legacy=0x7effb006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effadd9b700 1 -- 172.21.15.113:0/1564264383 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb006f750 con 0x7effb006c2e0 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effadd9b700 1 --2- 172.21.15.113:0/1564264383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb006c2e0 0x7effb015b620 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7effa0009840 tx=0x7effa0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3aa3ee88d6af4c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effad59a700 1 -- 172.21.15.113:0/1564264383 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0011830 con 0x7effb006c2e0 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effad59a700 1 -- 172.21.15.113:0/1564264383 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa0011990 con 0x7effb006c2e0 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.380+0000 7effad59a700 1 -- 172.21.15.113:0/1564264383 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0011d20 con 0x7effb006c2e0 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 -- 172.21.15.113:0/1564264383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb006c2e0 msgr2=0x7effb015b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 --2- 172.21.15.113:0/1564264383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb006c2e0 0x7effb015b620 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7effa0009840 tx=0x7effa0009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 -- 172.21.15.113:0/1564264383 shutdown_connections 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 --2- 172.21.15.113:0/1564264383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb006c2e0 0x7effb015b620 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 -- 172.21.15.113:0/1564264383 >> 172.21.15.113:0/1564264383 conn(0x7effb00d0c00 msgr2=0x7effb00d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:41.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 -- 172.21.15.113:0/1564264383 shutdown_connections 2024-01-16T13:52:41.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.381+0000 7effb4851700 1 -- 172.21.15.113:0/1564264383 wait complete. 2024-01-16T13:52:41.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.382+0000 7effb4851700 1 Processor -- start 2024-01-16T13:52:41.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.382+0000 7effb4851700 1 -- start start 2024-01-16T13:52:41.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effb4851700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb006c2e0 0x7effb01507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effb4851700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effb015b240 0x7effb00b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effb4851700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 0x7effb01f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effb4851700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb01f3850 con 0x7effb015bb60 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effb4851700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b1340 con 0x7effb006c2e0 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effb4851700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb00d0020 con 0x7effb015b240 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 0x7effb01f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 0x7effb01f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56204/0 (socket says 172.21.15.113:56204) 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 -- 172.21.15.113:0/3968794361 learned_addr learned my addr 172.21.15.113:0/3968794361 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effadd9b700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effb015b240 0x7effb00b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effae59c700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb006c2e0 0x7effb01507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 -- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effb015b240 msgr2=0x7effb00b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effb015b240 0x7effb00b0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 -- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb006c2e0 msgr2=0x7effb01507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb006c2e0 0x7effb01507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.383+0000 7effaed9d700 1 -- 172.21.15.113:0/3968794361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa00094a0 con 0x7effb015bb60 2024-01-16T13:52:41.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.384+0000 7effaed9d700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 0x7effb01f4870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7effa800cc50 tx=0x7effa800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.384+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa800ad80 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.384+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa800ce50 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.384+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb00aef50 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.384+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8003c40 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.384+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb00af510 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.385+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb01f39c0 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.385+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7effa80050d0 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.385+0000 7effa6ffd700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff9006b350 0x7eff9006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.385+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effa8089cc0 con 0x7effb015bb60 2024-01-16T13:52:41.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.385+0000 7effae59c700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff9006b350 0x7eff9006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.386+0000 7effae59c700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff9006b350 0x7eff9006d810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff980011e0 tx=0x7eff98009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.390+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effb01f39c0 con 0x7effb015bb60 2024-01-16T13:52:41.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.578+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7effb01f39c0 con 0x7effb015bb60 2024-01-16T13:52:41.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.579+0000 7effa6ffd700 1 -- 172.21.15.113:0/3968794361 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7effb01f39c0 con 0x7effb015bb60 2024-01-16T13:52:41.563 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff9006b350 msgr2=0x7eff9006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff9006b350 0x7eff9006d810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff980011e0 tx=0x7eff98009040 comp rx=0 tx=0).stop 2024-01-16T13:52:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 msgr2=0x7effb01f4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 0x7effb01f4870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7effa800cc50 tx=0x7effa800a340 comp rx=0 tx=0).stop 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 shutdown_connections 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7eff9006b350 0x7eff9006d810 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effb015bb60 0x7effb01f4870 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb006c2e0 0x7effb01507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 --2- 172.21.15.113:0/3968794361 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effb015b240 0x7effb00b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 >> 172.21.15.113:0/3968794361 conn(0x7effb00d0c00 msgr2=0x7effb006e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 shutdown_connections 2024-01-16T13:52:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.581+0000 7effb4851700 1 -- 172.21.15.113:0/3968794361 wait complete. 2024-01-16T13:52:41.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T13:52:41.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.715+0000 7f5896a1d700 1 Processor -- start 2024-01-16T13:52:41.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.715+0000 7f5896a1d700 1 -- start start 2024-01-16T13:52:41.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.716+0000 7f5896a1d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 0x7f58900d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.716+0000 7f5896a1d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589015bf80 con 0x7f589015b760 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.716+0000 7f5896a1d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589015bc80 con 0x7f58900b3830 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.716+0000 7f5896a1d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58900d0af0 con 0x7f589015b380 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.716+0000 7f588f7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f589015b760 0x7f58900b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38798/0 (socket says 172.21.15.113:38798) 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.716+0000 7f588f7fe700 1 -- 172.21.15.113:0/2027467424 learned_addr learned my addr 172.21.15.113:0/2027467424 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035202724 0 0) 0x7f589015bf80 con 0x7f589015b760 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 0x7f58900d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58780035b0 con 0x7f589015b760 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2106288504 0 0) 0x7f58780035b0 con 0x7f589015b760 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 msgr2=0x7f58900d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 0x7f58900d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 >> v1:172.21.15.124:6789/0 conn(0x7f58900b3830 legacy=0x7f589015c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900d0dd0 con 0x7f589015b760 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f5884001670 con 0x7f589015b760 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 >> v1:172.21.15.113:6789/0 conn(0x7f589015b760 legacy=0x7f58900b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5878003a30 0x7f5878003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5878004850 0x7f5878004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 0x7f5878005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58900d0af0 con 0x7f5878003a30 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587800bb40 con 0x7f5878004850 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587800bd00 con 0x7f589015b380 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5878003a30 0x7f5878003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588f7fe700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5878004850 0x7f5878004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f5894fba700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 0x7f5878005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 -- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 msgr2=0x7f5878005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f589015b380 0x7f5878005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 -- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5878004850 msgr2=0x7f5878004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5878004850 0x7f5878004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.717+0000 7f588ffff700 1 -- 172.21.15.113:0/2027467424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58780035b0 con 0x7f5878003a30 2024-01-16T13:52:41.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f588ffff700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5878003a30 0x7f5878003e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5880009b60 tx=0x7f5878005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b887bb5b4c1ec56c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588000e6c0 con 0x7f5878003a30 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588000e820 con 0x7f5878003a30 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f588effd700 1 -- 172.21.15.113:0/2027467424 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588000eb90 con 0x7f5878003a30 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5896a1d700 1 -- 172.21.15.113:0/2027467424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5878003a30 msgr2=0x7f5878003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5896a1d700 1 --2- 172.21.15.113:0/2027467424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5878003a30 0x7f5878003e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5880009b60 tx=0x7f5878005880 comp rx=0 tx=0).stop 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5894fba700 1 -- 172.21.15.113:0/2027467424 reap_dead start 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5896a1d700 1 -- 172.21.15.113:0/2027467424 shutdown_connections 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5896a1d700 1 -- 172.21.15.113:0/2027467424 >> 172.21.15.113:0/2027467424 conn(0x7f58900b5640 msgr2=0x7f58900b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5896a1d700 1 -- 172.21.15.113:0/2027467424 shutdown_connections 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.718+0000 7f5896a1d700 1 -- 172.21.15.113:0/2027467424 wait complete. 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.719+0000 7f5896a1d700 1 Processor -- start 2024-01-16T13:52:41.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 -- start start 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5878003a30 0x7f58901548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 0x7f5890147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58901530a0 0x7f589014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58901544c0 con 0x7f58901485b0 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589014e5b0 con 0x7f58901530a0 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f5896a1d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5890153840 con 0x7f5878003a30 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f588f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 0x7f5890147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f588f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 0x7f5890147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56218/0 (socket says 172.21.15.113:56218) 2024-01-16T13:52:41.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f588f7fe700 1 -- 172.21.15.113:0/3999080105 learned_addr learned my addr 172.21.15.113:0/3999080105 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.720+0000 7f588ffff700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5878003a30 0x7f58901548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f5894fba700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58901530a0 0x7f589014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588f7fe700 1 -- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5878003a30 msgr2=0x7f58901548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588f7fe700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5878003a30 0x7f58901548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588f7fe700 1 -- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58901530a0 msgr2=0x7f589014d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588f7fe700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58901530a0 0x7f589014d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588f7fe700 1 -- 172.21.15.113:0/3999080105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5890153b20 con 0x7f58901485b0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f5894fba700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58901530a0 0x7f589014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588f7fe700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 0x7f5890147f20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58840072f0 tx=0x7f5884007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58840077e0 con 0x7f58901485b0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400f8d0 con 0x7f58901485b0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400fc80 con 0x7f58901485b0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5890153e00 con 0x7f58901485b0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.721+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5890153550 con 0x7f58901485b0 2024-01-16T13:52:41.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.722+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5890154600 con 0x7f58901485b0 2024-01-16T13:52:41.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.722+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f588401d460 con 0x7f58901485b0 2024-01-16T13:52:41.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.723+0000 7f588cff9700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f587c06b350 0x7f587c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:41.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.723+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5884089af0 con 0x7f58901485b0 2024-01-16T13:52:41.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.723+0000 7f588ffff700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f587c06b350 0x7f587c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:41.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.723+0000 7f588ffff700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f587c06b350 0x7f587c06d810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58800090b0 tx=0x7f588001f040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:41.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.727+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5890154600 con 0x7f58901485b0 2024-01-16T13:52:41.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.931+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5890154600 con 0x7f58901485b0 2024-01-16T13:52:41.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.931+0000 7f588cff9700 1 -- 172.21.15.113:0/3999080105 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5890154600 con 0x7f58901485b0 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.933+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f587c06b350 msgr2=0x7f587c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.933+0000 7f5896a1d700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f587c06b350 0x7f587c06d810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58800090b0 tx=0x7f588001f040 comp rx=0 tx=0).stop 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.933+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 msgr2=0x7f5890147f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.933+0000 7f5896a1d700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 0x7f5890147f20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58840072f0 tx=0x7f5884007600 comp rx=0 tx=0).stop 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 shutdown_connections 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f587c06b350 0x7f587c06d810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58901530a0 0x7f589014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5878003a30 0x7f58901548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 --2- 172.21.15.113:0/3999080105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58901485b0 0x7f5890147f20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 >> 172.21.15.113:0/3999080105 conn(0x7f58900b5640 msgr2=0x7f58901594f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 shutdown_connections 2024-01-16T13:52:41.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:41.934+0000 7f5896a1d700 1 -- 172.21.15.113:0/3999080105 wait complete. 2024-01-16T13:52:41.918 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-16T13:52:41.929 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-01-16T13:52:41.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:41.929 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-01-16T13:52:41.964 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi113.front.sepia.ceph.com 2024-01-16T13:52:41.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:41.965 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T13:52:42.022 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:42.022 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:42.043 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:42.043 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:42.104 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:52:42.104 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:52:42.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:42.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.305+0000 7f1ea6273700 1 Processor -- start 2024-01-16T13:52:42.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.305+0000 7f1ea6273700 1 -- start start 2024-01-16T13:52:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.305+0000 7f1ea6273700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1ea015e080 0x7f1ea00cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.305+0000 7f1ea6273700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00d0150 con 0x7f1ea00d22f0 2024-01-16T13:52:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.305+0000 7f1ea6273700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00cee50 con 0x7f1ea00ce4d0 2024-01-16T13:52:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.305+0000 7f1ea6273700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea015e460 con 0x7f1ea015e080 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1ea4810700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1ea00d22f0 0x7f1ea00d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38810/0 (socket says 172.21.15.113:38810) 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1ea4810700 1 -- 172.21.15.113:0/816768386 learned_addr learned my addr 172.21.15.113:0/816768386 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9ffff700 1 --2- 172.21.15.113:0/816768386 >> v2:172.21.15.124:3300/0 conn(0x7f1ea015e080 0x7f1ea00cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916732752 0 0) 0x7f1ea00d0150 con 0x7f1ea00d22f0 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e880035b0 con 0x7f1ea00d22f0 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3864125534 0 0) 0x7f1e880035b0 con 0x7f1ea00d22f0 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 >> v2:172.21.15.124:3300/0 conn(0x7f1ea015e080 msgr2=0x7f1ea00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 --2- 172.21.15.113:0/816768386 >> v2:172.21.15.124:3300/0 conn(0x7f1ea015e080 0x7f1ea00cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 >> v1:172.21.15.124:6789/0 conn(0x7f1ea00ce4d0 legacy=0x7f1ea00d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00cf130 con 0x7f1ea00d22f0 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f1e980019f0 con 0x7f1ea00d22f0 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 >> v1:172.21.15.113:6789/0 conn(0x7f1ea00d22f0 legacy=0x7f1ea00d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e88003a00 0x7f1e88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e88004850 0x7f1e88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1e8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea015e460 con 0x7f1e88005210 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8800c0b0 con 0x7f1e88003a00 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8800c270 con 0x7f1e88004850 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1ea4810700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1e8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9ffff700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e88003a00 0x7f1e88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.306+0000 7f1e9f7fe700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e88004850 0x7f1e88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 -- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e88004850 msgr2=0x7f1e88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e88004850 0x7f1e88004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 -- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e88003a00 msgr2=0x7f1e88003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e88003a00 0x7f1e88003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 -- 172.21.15.113:0/816768386 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880035b0 con 0x7f1e88005210 2024-01-16T13:52:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 -- 172.21.15.113:0/816768386 reap_dead start 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea4810700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1e8800ba10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800a830 tx=0x7f1e88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b04ca0fc2de808 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9800a2f0 con 0x7f1e88005210 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e98012620 con 0x7f1e88005210 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1e9effd700 1 -- 172.21.15.113:0/816768386 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e980129d0 con 0x7f1e88005210 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea6273700 1 -- 172.21.15.113:0/816768386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 msgr2=0x7f1e8800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.307+0000 7f1ea6273700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1e8800ba10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800a830 tx=0x7f1e88005880 comp rx=0 tx=0).stop 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.308+0000 7f1ea6273700 1 -- 172.21.15.113:0/816768386 shutdown_connections 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.308+0000 7f1ea6273700 1 --2- 172.21.15.113:0/816768386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1e8800ba10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.308+0000 7f1ea6273700 1 -- 172.21.15.113:0/816768386 >> 172.21.15.113:0/816768386 conn(0x7f1ea0073130 msgr2=0x7f1ea009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.308+0000 7f1ea6273700 1 -- 172.21.15.113:0/816768386 shutdown_connections 2024-01-16T13:52:42.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.308+0000 7f1ea6273700 1 -- 172.21.15.113:0/816768386 wait complete. 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.309+0000 7f1ea6273700 1 Processor -- start 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.309+0000 7f1ea6273700 1 -- start start 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1ea6273700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1ea01f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1ea6273700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ea00b6ac0 0x7f1ea00b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1ea6273700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ea0159540 0x7f1ea01599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1ea6273700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea00b7480 con 0x7f1e88005210 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1ea6273700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea00c5d70 con 0x7f1ea0159540 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1ea6273700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea00c5fc0 con 0x7f1ea00b6ac0 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1ea01f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1ea01f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56250/0 (socket says 172.21.15.113:56250) 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 -- 172.21.15.113:0/1469352678 learned_addr learned my addr 172.21.15.113:0/1469352678 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 -- 172.21.15.113:0/1469352678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ea00b6ac0 msgr2=0x7f1ea00b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ea00b6ac0 0x7f1ea00b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 -- 172.21.15.113:0/1469352678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ea0159540 msgr2=0x7f1ea01599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ea0159540 0x7f1ea01599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.310+0000 7f1e9ffff700 1 -- 172.21.15.113:0/1469352678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00c62a0 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.311+0000 7f1e9ffff700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1ea01f3cf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1e900119d0 tx=0x7f1e90011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.311+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000f940 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.311+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9000faa0 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.311+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea00c6580 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.311+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000fe30 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.311+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea00c6b40 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.312+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea0067f60 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.312+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1e90014440 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.313+0000 7f1e9cff9700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e8c06b350 0x7f1e8c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.313+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e9008a2a0 con 0x7f1e88005210 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.313+0000 7f1e9f7fe700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e8c06b350 0x7f1e8c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.313+0000 7f1e9f7fe700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e8c06b350 0x7f1e8c06d810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e94000c00 tx=0x7f1e94009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.317+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e90055110 con 0x7f1e88005210 2024-01-16T13:52:42.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.509+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ea00d19c0 con 0x7f1e88005210 2024-01-16T13:52:42.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.510+0000 7f1e9cff9700 1 -- 172.21.15.113:0/1469352678 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f1e90058730 con 0x7f1e88005210 2024-01-16T13:52:42.494 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:42.494 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.512+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e8c06b350 msgr2=0x7f1e8c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.512+0000 7f1ea6273700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e8c06b350 0x7f1e8c06d810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e94000c00 tx=0x7f1e94009040 comp rx=0 tx=0).stop 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.512+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 msgr2=0x7f1ea01f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.512+0000 7f1ea6273700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1ea01f3cf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1e900119d0 tx=0x7f1e90011d90 comp rx=0 tx=0).stop 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 shutdown_connections 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e8c06b350 0x7f1e8c06d810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ea0159540 0x7f1ea01599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e88005210 0x7f1ea01f3cf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 --2- 172.21.15.113:0/1469352678 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ea00b6ac0 0x7f1ea00b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 >> 172.21.15.113:0/1469352678 conn(0x7f1ea0073130 msgr2=0x7f1ea015f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:42.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 shutdown_connections 2024-01-16T13:52:42.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.513+0000 7f1ea6273700 1 -- 172.21.15.113:0/1469352678 wait complete. 2024-01-16T13:52:42.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:42.508 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.651+0000 7fa843d29700 1 Processor -- start 2024-01-16T13:52:42.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.651+0000 7fa843d29700 1 -- start start 2024-01-16T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa843d29700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa83c0cffc0 0x7fa83c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa843d29700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa83c15f8d0 0x7fa83c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa843d29700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83c0b3860 con 0x7fa83c0d03a0 2024-01-16T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa843d29700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c0b2520 con 0x7fa83c0cffc0 2024-01-16T13:52:42.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa843d29700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c0b2800 con 0x7fa83c15f8d0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa841ac5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa83c0cffc0 0x7fa83c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa8412c4700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa83c0d03a0 0x7fa83c15f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:38818/0 (socket says 172.21.15.113:38818) 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa8412c4700 1 -- 172.21.15.113:0/3203666565 learned_addr learned my addr 172.21.15.113:0/3203666565 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa8422c6700 1 --2- 172.21.15.113:0/3203666565 >> v2:172.21.15.124:3300/0 conn(0x7fa83c15f8d0 0x7fa83c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa841ac5700 1 -- 172.21.15.113:0/3203666565 >> v2:172.21.15.124:3300/0 conn(0x7fa83c15f8d0 msgr2=0x7fa83c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa841ac5700 1 --2- 172.21.15.113:0/3203666565 >> v2:172.21.15.124:3300/0 conn(0x7fa83c15f8d0 0x7fa83c0ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa841ac5700 1 -- 172.21.15.113:0/3203666565 >> v1:172.21.15.113:6789/0 conn(0x7fa83c0d03a0 legacy=0x7fa83c15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa840ac3700 1 -- 172.21.15.113:0/3203666565 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704680991 0 0) 0x7fa82c001110 con 0x7fa83c0d03a0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa841ac5700 1 -- 172.21.15.113:0/3203666565 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c0b2ae0 con 0x7fa83c0cffc0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa8422c6700 1 --2- 172.21.15.113:0/3203666565 >> v2:172.21.15.124:3300/0 conn(0x7fa83c15f8d0 0x7fa83c0ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.652+0000 7fa841ac5700 1 --2- 172.21.15.113:0/3203666565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c0cffc0 0x7fa83c0b3320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0d0db0 tx=0x7fa838009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367bbcc5915e5cc6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa840ac3700 1 -- 172.21.15.113:0/3203666565 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838011830 con 0x7fa83c0cffc0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa840ac3700 1 -- 172.21.15.113:0/3203666565 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838011990 con 0x7fa83c0cffc0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa840ac3700 1 -- 172.21.15.113:0/3203666565 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838011d20 con 0x7fa83c0cffc0 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 -- 172.21.15.113:0/3203666565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c0cffc0 msgr2=0x7fa83c0b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 --2- 172.21.15.113:0/3203666565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c0cffc0 0x7fa83c0b3320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0d0db0 tx=0x7fa838009b20 comp rx=0 tx=0).stop 2024-01-16T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 -- 172.21.15.113:0/3203666565 shutdown_connections 2024-01-16T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 --2- 172.21.15.113:0/3203666565 >> v2:172.21.15.124:3300/0 conn(0x7fa83c15f8d0 0x7fa83c0ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 --2- 172.21.15.113:0/3203666565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c0cffc0 0x7fa83c0b3320 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 -- 172.21.15.113:0/3203666565 >> 172.21.15.113:0/3203666565 conn(0x7fa83c0d1e50 msgr2=0x7fa83c0d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 -- 172.21.15.113:0/3203666565 shutdown_connections 2024-01-16T13:52:42.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.653+0000 7fa843d29700 1 -- 172.21.15.113:0/3203666565 wait complete. 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.654+0000 7fa843d29700 1 Processor -- start 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.654+0000 7fa843d29700 1 -- start start 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa843d29700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa83c0cffc0 0x7fa83c0c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa843d29700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa83c0d03a0 0x7fa83c0b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa843d29700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 0x7fa83c0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa843d29700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0b5710 con 0x7fa83c15f8d0 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa843d29700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0b6e50 con 0x7fa83c0d03a0 2024-01-16T13:52:42.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa843d29700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c06c830 con 0x7fa83c0cffc0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa8422c6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 0x7fa83c0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa8422c6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 0x7fa83c0b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56264/0 (socket says 172.21.15.113:56264) 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa8422c6700 1 -- 172.21.15.113:0/2211605527 learned_addr learned my addr 172.21.15.113:0/2211605527 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa841ac5700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa83c0cffc0 0x7fa83c0c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.655+0000 7fa8412c4700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa83c0d03a0 0x7fa83c0b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8422c6700 1 -- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa83c0cffc0 msgr2=0x7fa83c0c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8422c6700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa83c0cffc0 0x7fa83c0c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8422c6700 1 -- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa83c0d03a0 msgr2=0x7fa83c0b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8422c6700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa83c0d03a0 0x7fa83c0b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8422c6700 1 -- 172.21.15.113:0/2211605527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8380094a0 con 0x7fa83c15f8d0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa841ac5700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa83c0cffc0 0x7fa83c0c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8422c6700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 0x7fa83c0b77e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa834007e40 tx=0x7fa83400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83400de90 con 0x7fa83c15f8d0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83400f040 con 0x7fa83c15f8d0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83c06cb10 con 0x7fa83c15f8d0 2024-01-16T13:52:42.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834003c20 con 0x7fa83c15f8d0 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.656+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83c0b53a0 con 0x7fa83c15f8d0 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.657+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83c0b58c0 con 0x7fa83c15f8d0 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.657+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa834004030 con 0x7fa83c15f8d0 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.658+0000 7fa8327fc700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa82006b350 0x7fa82006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.658+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa834089c60 con 0x7fa83c15f8d0 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.658+0000 7fa841ac5700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa82006b350 0x7fa82006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.658+0000 7fa841ac5700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa82006b350 0x7fa82006d810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa8380090b0 tx=0x7fa838008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.662+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa83c0b58c0 con 0x7fa83c15f8d0 2024-01-16T13:52:42.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.852+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa83c069020 con 0x7fa83c15f8d0 2024-01-16T13:52:42.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.852+0000 7fa8327fc700 1 -- 172.21.15.113:0/2211605527 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa83401a0a0 con 0x7fa83c15f8d0 2024-01-16T13:52:42.835 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa82006b350 msgr2=0x7fa82006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa82006b350 0x7fa82006d810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa8380090b0 tx=0x7fa838008030 comp rx=0 tx=0).stop 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 msgr2=0x7fa83c0b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 0x7fa83c0b77e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa834007e40 tx=0x7fa83400d3b0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 shutdown_connections 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa82006b350 0x7fa82006d810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa83c15f8d0 0x7fa83c0b77e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa83c0cffc0 0x7fa83c0c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 --2- 172.21.15.113:0/2211605527 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa83c0d03a0 0x7fa83c0b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 >> 172.21.15.113:0/2211605527 conn(0x7fa83c0d1e50 msgr2=0x7fa83c15dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 shutdown_connections 2024-01-16T13:52:42.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.854+0000 7fa843d29700 1 -- 172.21.15.113:0/2211605527 wait complete. 2024-01-16T13:52:42.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.990+0000 7ff7d7e38700 1 Processor -- start 2024-01-16T13:52:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 -- start start 2024-01-16T13:52:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d006e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff7d006e860 0x7ff7d0074000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff7d0074540 0x7ff7d0071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d015f2f0 con 0x7ff7d0074540 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d015e480 con 0x7ff7d006e860 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.991+0000 7ff7d7e38700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d006ece0 con 0x7ff7d0158570 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff7d0074540 0x7ff7d0071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff7d0074540 0x7ff7d0071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56280/0 (socket says 172.21.15.113:56280) 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 -- 172.21.15.113:0/3119057485 learned_addr learned my addr 172.21.15.113:0/3119057485 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d5bd4700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d53d3700 1 --2- 172.21.15.113:0/3119057485 >> v2:172.21.15.124:3300/0 conn(0x7ff7d006e860 0x7ff7d0074000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 -- 172.21.15.113:0/3119057485 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 msgr2=0x7ff7d006e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d006e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 -- 172.21.15.113:0/3119057485 >> v2:172.21.15.124:3300/0 conn(0x7ff7d006e860 msgr2=0x7ff7d0074000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 --2- 172.21.15.113:0/3119057485 >> v2:172.21.15.124:3300/0 conn(0x7ff7d006e860 0x7ff7d0074000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 -- 172.21.15.113:0/3119057485 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d0158950 con 0x7ff7d0074540 2024-01-16T13:52:42.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d5bd4700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d006e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d53d3700 1 --2- 172.21.15.113:0/3119057485 >> v2:172.21.15.124:3300/0 conn(0x7ff7d006e860 0x7ff7d0074000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d63d5700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d0071b90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0062a60 tx=0x7ff7c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cd55c4d63f5368 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d4bd2700 1 -- 172.21.15.113:0/3119057485 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800e070 con 0x7ff7d0074540 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d4bd2700 1 -- 172.21.15.113:0/3119057485 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c800b9c0 con 0x7ff7d0074540 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.992+0000 7ff7d4bd2700 1 -- 172.21.15.113:0/3119057485 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800bd70 con 0x7ff7d0074540 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3119057485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 msgr2=0x7ff7d0071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d0071b90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0062a60 tx=0x7ff7c800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3119057485 shutdown_connections 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d0071b90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3119057485 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d006e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3119057485 >> v2:172.21.15.124:3300/0 conn(0x7ff7d006e860 0x7ff7d0074000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3119057485 >> 172.21.15.113:0/3119057485 conn(0x7ff7d00cd6b0 msgr2=0x7ff7d00cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:42.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3119057485 shutdown_connections 2024-01-16T13:52:42.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.993+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3119057485 wait complete. 2024-01-16T13:52:42.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.994+0000 7ff7d7e38700 1 Processor -- start 2024-01-16T13:52:42.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.994+0000 7ff7d7e38700 1 -- start start 2024-01-16T13:52:42.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d7e38700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7d006e860 0x7ff7d01f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d7e38700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d01536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d7e38700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d0158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d7e38700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d015a650 con 0x7ff7d0074540 2024-01-16T13:52:42.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d7e38700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01595e0 con 0x7ff7d006e860 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d7e38700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d01f7060 con 0x7ff7d0158570 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d01536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d01536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56288/0 (socket says 172.21.15.113:56288) 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 -- 172.21.15.113:0/3980172909 learned_addr learned my addr 172.21.15.113:0/3980172909 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d5bd4700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7d006e860 0x7ff7d01f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d63d5700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d0158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 -- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 msgr2=0x7ff7d0158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d0158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 -- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7d006e860 msgr2=0x7ff7d01f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7d006e860 0x7ff7d01f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 -- 172.21.15.113:0/3980172909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8009b30 con 0x7ff7d0074540 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.995+0000 7ff7d53d3700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d01536a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00d800 tx=0x7ff7cc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.996+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc004030 con 0x7ff7d0074540 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.996+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc004190 con 0x7ff7d0074540 2024-01-16T13:52:42.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.996+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d01f7340 con 0x7ff7d0074540 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.996+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc004520 con 0x7ff7d0074540 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.996+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d0153be0 con 0x7ff7d0074540 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.997+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0069020 con 0x7ff7d0074540 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.997+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff7cc0036a0 con 0x7ff7d0074540 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.997+0000 7ff7c67fc700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff7b406b350 0x7ff7b406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.997+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7cc08a0f0 con 0x7ff7d0074540 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.997+0000 7ff7d5bd4700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff7b406b350 0x7ff7b406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:42.998+0000 7ff7d5bd4700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff7b406b350 0x7ff7b406d810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000c900 tx=0x7ff7c0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:42.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.002+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7cc054f60 con 0x7ff7d0074540 2024-01-16T13:52:43.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.189+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff7d00ce440 con 0x7ff7d0074540 2024-01-16T13:52:43.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.189+0000 7ff7c67fc700 1 -- 172.21.15.113:0/3980172909 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff7cc058580 con 0x7ff7d0074540 2024-01-16T13:52:43.173 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff7b406b350 msgr2=0x7ff7b406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff7b406b350 0x7ff7b406d810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000c900 tx=0x7ff7c0008040 comp rx=0 tx=0).stop 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 msgr2=0x7ff7d01536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d01536a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00d800 tx=0x7ff7cc00dbc0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 shutdown_connections 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff7b406b350 0x7ff7b406d810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7d0158570 0x7ff7d0158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7d006e860 0x7ff7d01f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 --2- 172.21.15.113:0/3980172909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7d0074540 0x7ff7d01536a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.191+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 >> 172.21.15.113:0/3980172909 conn(0x7ff7d00cd6b0 msgr2=0x7ff7d0156350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:43.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.192+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 shutdown_connections 2024-01-16T13:52:43.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.192+0000 7ff7d7e38700 1 -- 172.21.15.113:0/3980172909 wait complete. 2024-01-16T13:52:43.186 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:43.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.327+0000 7f1880b8a700 1 Processor -- start 2024-01-16T13:52:43.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.327+0000 7f1880b8a700 1 -- start start 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1880b8a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187c06c2e0 0x7f187c15b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1880b8a700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c06eeb0 con 0x7f187c15bb60 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1880b8a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c06f190 con 0x7f187c15b240 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1880b8a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187c06f470 con 0x7f187c06c2e0 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1879d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187c06c2e0 0x7f187c15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f187ad9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f187c15bb60 0x7f187c1647f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53418/0 (socket says 172.21.15.113:53418) 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f187ad9d700 1 -- 172.21.15.113:0/1302933706 learned_addr learned my addr 172.21.15.113:0/1302933706 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1879d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187c06c2e0 0x7f187c15b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55944/0 (socket says 172.21.15.113:55944) 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1879d9b700 1 -- 172.21.15.113:0/1302933706 >> v1:172.21.15.124:6789/0 conn(0x7f187c15b240 legacy=0x7f187c06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1879d9b700 1 -- 172.21.15.113:0/1302933706 >> v1:172.21.15.113:6789/0 conn(0x7f187c15bb60 legacy=0x7f187c1647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1879d9b700 1 -- 172.21.15.113:0/1302933706 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c06f750 con 0x7f187c06c2e0 2024-01-16T13:52:43.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.328+0000 7f1879d9b700 1 --2- 172.21.15.113:0/1302933706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c06c2e0 0x7f187c15b620 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f186c009840 tx=0x7f186c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3524912012d60d0f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:43.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f187959a700 1 -- 172.21.15.113:0/1302933706 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c011830 con 0x7f187c06c2e0 2024-01-16T13:52:43.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f187959a700 1 -- 172.21.15.113:0/1302933706 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c011990 con 0x7f187c06c2e0 2024-01-16T13:52:43.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f187959a700 1 -- 172.21.15.113:0/1302933706 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c011d20 con 0x7f187c06c2e0 2024-01-16T13:52:43.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 -- 172.21.15.113:0/1302933706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c06c2e0 msgr2=0x7f187c15b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 --2- 172.21.15.113:0/1302933706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c06c2e0 0x7f187c15b620 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f186c009840 tx=0x7f186c009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 -- 172.21.15.113:0/1302933706 shutdown_connections 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 --2- 172.21.15.113:0/1302933706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c06c2e0 0x7f187c15b620 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 -- 172.21.15.113:0/1302933706 >> 172.21.15.113:0/1302933706 conn(0x7f187c0d0c00 msgr2=0x7f187c0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 -- 172.21.15.113:0/1302933706 shutdown_connections 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.329+0000 7f1880b8a700 1 -- 172.21.15.113:0/1302933706 wait complete. 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 Processor -- start 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 -- start start 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187c06c2e0 0x7f187c1edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 0x7f187c0afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f187c15bb60 0x7f187c1fc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c1fc670 con 0x7f187c15b240 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c1fbca0 con 0x7f187c15bb60 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.331+0000 7f1880b8a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0b08e0 con 0x7f187c06c2e0 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 0x7f187c0afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 0x7f187c0afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55958/0 (socket says 172.21.15.113:55958) 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 -- 172.21.15.113:0/2501486625 learned_addr learned my addr 172.21.15.113:0/2501486625 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f187a59c700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187c06c2e0 0x7f187c1edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f187ad9d700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f187c15bb60 0x7f187c1fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 -- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187c06c2e0 msgr2=0x7f187c1edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187c06c2e0 0x7f187c1edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 -- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f187c15bb60 msgr2=0x7f187c1fc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f187c15bb60 0x7f187c1fc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 -- 172.21.15.113:0/2501486625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c0094a0 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f187ad9d700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f187c15bb60 0x7f187c1fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f187a59c700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187c06c2e0 0x7f187c1edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1879d9b700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 0x7f187c0afc20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f186c000c00 tx=0x7f186c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c0038a0 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c01e070 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187c0b0320 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c003c60 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.332+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187c1fb930 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.333+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187c1fb790 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.334+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f186c02d450 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.334+0000 7f186affd700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185c06b380 0x7f185c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.334+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186c08ac20 con 0x7f187c15b240 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.334+0000 7f187a59c700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185c06b380 0x7f185c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.334+0000 7f187a59c700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185c06b380 0x7f185c06d840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1870000f60 tx=0x7f1870009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:43.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.339+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f187c1fb790 con 0x7f187c15b240 2024-01-16T13:52:43.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.530+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f187c1fb790 con 0x7f187c15b240 2024-01-16T13:52:43.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.531+0000 7f186affd700 1 -- 172.21.15.113:0/2501486625 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f187c1fb790 con 0x7f187c15b240 2024-01-16T13:52:43.514 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:43.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185c06b380 msgr2=0x7f185c06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185c06b380 0x7f185c06d840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1870000f60 tx=0x7f1870009040 comp rx=0 tx=0).stop 2024-01-16T13:52:43.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 msgr2=0x7f187c0afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 0x7f187c0afc20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f186c000c00 tx=0x7f186c004750 comp rx=0 tx=0).stop 2024-01-16T13:52:43.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 shutdown_connections 2024-01-16T13:52:43.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185c06b380 0x7f185c06d840 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f187c15bb60 0x7f187c1fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187c06c2e0 0x7f187c1edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 --2- 172.21.15.113:0/2501486625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187c15b240 0x7f187c0afc20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 >> 172.21.15.113:0/2501486625 conn(0x7f187c0d0c00 msgr2=0x7f187c06e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:43.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 shutdown_connections 2024-01-16T13:52:43.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.533+0000 7f1880b8a700 1 -- 172.21.15.113:0/2501486625 wait complete. 2024-01-16T13:52:43.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T13:52:43.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.668+0000 7f08811c1700 1 Processor -- start 2024-01-16T13:52:43.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.668+0000 7f08811c1700 1 -- start start 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f08811c1700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f087c15e080 0x7f087c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f08811c1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f087c0ce4d0 0x7f087c0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f08811c1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce910 0x7f087c0d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f08811c1700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c15e460 con 0x7f087c15e080 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f08811c1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c0d0150 con 0x7f087c0ce4d0 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f08811c1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cee50 con 0x7f087c0ce910 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f087c15e080 0x7f087c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f087c15e080 0x7f087c0cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55966/0 (socket says 172.21.15.113:55966) 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 -- 172.21.15.113:0/4280546606 learned_addr learned my addr 172.21.15.113:0/4280546606 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:43.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087b59e700 1 --2- 172.21.15.113:0/4280546606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce910 0x7f087c0d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087a59c700 1 --2- 172.21.15.113:0/4280546606 >> v2:172.21.15.124:3300/0 conn(0x7f087c0ce4d0 0x7f087c0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 -- 172.21.15.113:0/4280546606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce910 msgr2=0x7f087c0d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 --2- 172.21.15.113:0/4280546606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce910 0x7f087c0d2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 -- 172.21.15.113:0/4280546606 >> v2:172.21.15.124:3300/0 conn(0x7f087c0ce4d0 msgr2=0x7f087c0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 --2- 172.21.15.113:0/4280546606 >> v2:172.21.15.124:3300/0 conn(0x7f087c0ce4d0 0x7f087c0d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 -- 172.21.15.113:0/4280546606 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0cf130 con 0x7f087c15e080 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f087ad9d700 1 --2- 172.21.15.113:0/4280546606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c15e080 0x7f087c0cc2a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087c0ce290 tx=0x7f0870009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbc9586850977ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.669+0000 7f0879d9b700 1 -- 172.21.15.113:0/4280546606 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000b870 con 0x7f087c15e080 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f0879d9b700 1 -- 172.21.15.113:0/4280546606 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087000b9d0 con 0x7f087c15e080 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f0879d9b700 1 -- 172.21.15.113:0/4280546606 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000bd60 con 0x7f087c15e080 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 -- 172.21.15.113:0/4280546606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c15e080 msgr2=0x7f087c0cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 --2- 172.21.15.113:0/4280546606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c15e080 0x7f087c0cc2a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087c0ce290 tx=0x7f0870009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 -- 172.21.15.113:0/4280546606 shutdown_connections 2024-01-16T13:52:43.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 --2- 172.21.15.113:0/4280546606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c15e080 0x7f087c0cc2a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 --2- 172.21.15.113:0/4280546606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce910 0x7f087c0d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 --2- 172.21.15.113:0/4280546606 >> v2:172.21.15.124:3300/0 conn(0x7f087c0ce4d0 0x7f087c0d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 -- 172.21.15.113:0/4280546606 >> 172.21.15.113:0/4280546606 conn(0x7f087c073130 msgr2=0x7f087c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:43.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 -- 172.21.15.113:0/4280546606 shutdown_connections 2024-01-16T13:52:43.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.670+0000 7f08811c1700 1 -- 172.21.15.113:0/4280546606 wait complete. 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 Processor -- start 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 -- start start 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce4d0 0x7f087c15d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 0x7f087c0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f087c15e080 0x7f087c0caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c15d7e0 con 0x7f087c0ce910 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0c9570 con 0x7f087c15e080 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f08811c1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0c9850 con 0x7f087c0ce4d0 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f087a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 0x7f087c0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f087a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 0x7f087c0c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55982/0 (socket says 172.21.15.113:55982) 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.672+0000 7f087a59c700 1 -- 172.21.15.113:0/232462304 learned_addr learned my addr 172.21.15.113:0/232462304 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087ad9d700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce4d0 0x7f087c15d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087b59e700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f087c15e080 0x7f087c0caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087a59c700 1 -- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce4d0 msgr2=0x7f087c15d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087a59c700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce4d0 0x7f087c15d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087a59c700 1 -- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f087c15e080 msgr2=0x7f087c0caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087a59c700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f087c15e080 0x7f087c0caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087a59c700 1 -- 172.21.15.113:0/232462304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0870009550 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f087a59c700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 0x7f087c0c8d20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e810 tx=0x7f086c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00c070 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c013610 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c0c9b30 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c0139c0 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.673+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c0ca0f0 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.674+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c067f60 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.674+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f086c013b20 con 0x7f087c0ce910 2024-01-16T13:52:43.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.675+0000 7f08637fe700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f086806b380 0x7f086806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:43.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.675+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f086c089960 con 0x7f087c0ce910 2024-01-16T13:52:43.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.675+0000 7f087ad9d700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f086806b380 0x7f086806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:43.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.675+0000 7f087ad9d700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f086806b380 0x7f086806d840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0870014040 tx=0x7f0870008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:43.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.679+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f086c054750 con 0x7f087c0ce910 2024-01-16T13:52:43.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.887+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f087c0d19c0 con 0x7f087c0ce910 2024-01-16T13:52:43.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.887+0000 7f08637fe700 1 -- 172.21.15.113:0/232462304 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f086c057d70 con 0x7f087c0ce910 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f086806b380 msgr2=0x7f086806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f086806b380 0x7f086806d840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0870014040 tx=0x7f0870008030 comp rx=0 tx=0).stop 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 msgr2=0x7f087c0c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 0x7f087c0c8d20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e810 tx=0x7f086c00eb20 comp rx=0 tx=0).stop 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 shutdown_connections 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f086806b380 0x7f086806d840 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f087c15e080 0x7f087c0caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f087c0ce4d0 0x7f087c15d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 --2- 172.21.15.113:0/232462304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f087c0ce910 0x7f087c0c8d20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:43.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 >> 172.21.15.113:0/232462304 conn(0x7f087c073130 msgr2=0x7f087c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:43.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 shutdown_connections 2024-01-16T13:52:43.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:43.890+0000 7f08811c1700 1 -- 172.21.15.113:0/232462304 wait complete. 2024-01-16T13:52:43.874 INFO:teuthology.orchestra.run.smithi113.stderr:listed 0 entries 2024-01-16T13:52:43.885 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-01-16T13:52:43.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:43.885 DEBUG:teuthology.orchestra.run.smithi124:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-01-16T13:52:43.932 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2024-01-16T13:52:43.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:52:43.932 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T13:52:43.953 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T13:52:43.953 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:44.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.091+0000 7f4146a4a700 1 Processor -- start 2024-01-16T13:52:44.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.091+0000 7f4146a4a700 1 -- start start 2024-01-16T13:52:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4146a4a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f41400ce090 0x7f414006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4146a4a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4140073f90 0x7f4140074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4146a4a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414006e7f0 con 0x7f414015e2a0 2024-01-16T13:52:44.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4146a4a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4140074960 con 0x7f4140073f90 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4146a4a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414006dec0 con 0x7f41400ce090 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4140073f90 0x7f4140074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4140073f90 0x7f4140074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55996/0 (socket says 172.21.15.113:55996) 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 -- 172.21.15.113:0/3100241084 learned_addr learned my addr 172.21.15.113:0/3100241084 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f413f7fe700 1 --2- 172.21.15.113:0/3100241084 >> v2:172.21.15.124:3300/0 conn(0x7f41400ce090 0x7f414006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 -- 172.21.15.113:0/3100241084 >> v2:172.21.15.124:3300/0 conn(0x7f41400ce090 msgr2=0x7f414006ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 --2- 172.21.15.113:0/3100241084 >> v2:172.21.15.124:3300/0 conn(0x7f41400ce090 0x7f414006ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 -- 172.21.15.113:0/3100241084 >> v1:172.21.15.124:6789/0 conn(0x7f414015e2a0 legacy=0x7f414006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 -- 172.21.15.113:0/3100241084 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414006e1a0 con 0x7f4140073f90 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f4144fe7700 1 --2- 172.21.15.113:0/3100241084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4140073f90 0x7f4140074420 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4140071840 tx=0x7f413800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a219fc19a7869f56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f413effd700 1 -- 172.21.15.113:0/3100241084 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413800e070 con 0x7f4140073f90 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f413effd700 1 -- 172.21.15.113:0/3100241084 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4138003a20 con 0x7f4140073f90 2024-01-16T13:52:44.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.092+0000 7f413effd700 1 -- 172.21.15.113:0/3100241084 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138003dd0 con 0x7f4140073f90 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 -- 172.21.15.113:0/3100241084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4140073f90 msgr2=0x7f4140074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 --2- 172.21.15.113:0/3100241084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4140073f90 0x7f4140074420 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4140071840 tx=0x7f413800b330 comp rx=0 tx=0).stop 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 -- 172.21.15.113:0/3100241084 shutdown_connections 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 --2- 172.21.15.113:0/3100241084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4140073f90 0x7f4140074420 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 --2- 172.21.15.113:0/3100241084 >> v2:172.21.15.124:3300/0 conn(0x7f41400ce090 0x7f414006ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 -- 172.21.15.113:0/3100241084 >> 172.21.15.113:0/3100241084 conn(0x7f41400cd570 msgr2=0x7f41400cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 -- 172.21.15.113:0/3100241084 shutdown_connections 2024-01-16T13:52:44.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.093+0000 7f4146a4a700 1 -- 172.21.15.113:0/3100241084 wait complete. 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.094+0000 7f4146a4a700 1 Processor -- start 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.094+0000 7f4146a4a700 1 -- start start 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4146a4a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4140073f90 0x7f4140145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4146a4a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41400ce090 0x7f4140149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4146a4a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 0x7f41400b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4146a4a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140146430 con 0x7f414015e2a0 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4146a4a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140149690 con 0x7f41400ce090 2024-01-16T13:52:44.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4146a4a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4140158450 con 0x7f4140073f90 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 0x7f41400b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 0x7f41400b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56014/0 (socket says 172.21.15.113:56014) 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 -- 172.21.15.113:0/1292883620 learned_addr learned my addr 172.21.15.113:0/1292883620 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 -- 172.21.15.113:0/1292883620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4140073f90 msgr2=0x7f4140145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4140073f90 0x7f4140145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 -- 172.21.15.113:0/1292883620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41400ce090 msgr2=0x7f4140149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41400ce090 0x7f4140149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.095+0000 7f4144fe7700 1 -- 172.21.15.113:0/1292883620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4138009b30 con 0x7f414015e2a0 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.096+0000 7f4144fe7700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 0x7f41400b17d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4138015040 tx=0x7f4138004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.096+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138004240 con 0x7f414015e2a0 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.096+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41380043a0 con 0x7f414015e2a0 2024-01-16T13:52:44.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.096+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4140158730 con 0x7f414015e2a0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.096+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138004750 con 0x7f414015e2a0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.096+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4140158cf0 con 0x7f414015e2a0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.097+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4140067f60 con 0x7f414015e2a0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.097+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4138019070 con 0x7f414015e2a0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.097+0000 7f413cff9700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f412c06b350 0x7f412c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.098+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f413808ab70 con 0x7f414015e2a0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.098+0000 7f413ffff700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f412c06b350 0x7f412c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.098+0000 7f413ffff700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f412c06b350 0x7f412c06d810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41400cd430 tx=0x7f4130007a40 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.102+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41380559f0 con 0x7f414015e2a0 2024-01-16T13:52:44.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.337+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4140072e00 con 0x7f414015e2a0 2024-01-16T13:52:44.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.337+0000 7f413cff9700 1 -- 172.21.15.113:0/1292883620 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f4138059010 con 0x7f414015e2a0 2024-01-16T13:52:44.320 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:44.320 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f412c06b350 msgr2=0x7f412c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f412c06b350 0x7f412c06d810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41400cd430 tx=0x7f4130007a40 comp rx=0 tx=0).stop 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 msgr2=0x7f41400b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 0x7f41400b17d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4138015040 tx=0x7f4138004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 shutdown_connections 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f412c06b350 0x7f412c06d810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f414015e2a0 0x7f41400b17d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4140073f90 0x7f4140145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 --2- 172.21.15.113:0/1292883620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41400ce090 0x7f4140149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 >> 172.21.15.113:0/1292883620 conn(0x7f41400cd570 msgr2=0x7f41400afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 shutdown_connections 2024-01-16T13:52:44.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.339+0000 7f4146a4a700 1 -- 172.21.15.113:0/1292883620 wait complete. 2024-01-16T13:52:44.324 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:44.335 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:44.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.473+0000 7f5a1fd54700 1 Processor -- start 2024-01-16T13:52:44.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.473+0000 7f5a1fd54700 1 -- start start 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1fd54700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a18140f20 0x7f5a180b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1fd54700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5a180b3900 0x7f5a180b2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1fd54700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a18161b40 con 0x7f5a180b2590 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1fd54700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a180b2a20 con 0x7f5a180b3900 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1fd54700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180b3e10 con 0x7f5a18140f20 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5a180b3900 0x7f5a180b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5a180b3900 0x7f5a180b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56018/0 (socket says 172.21.15.113:56018) 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 -- 172.21.15.113:0/2342475547 learned_addr learned my addr 172.21.15.113:0/2342475547 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1daf0700 1 --2- 172.21.15.113:0/2342475547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a18140f20 0x7f5a180b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 -- 172.21.15.113:0/2342475547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a18140f20 msgr2=0x7f5a180b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 --2- 172.21.15.113:0/2342475547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a18140f20 0x7f5a180b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 -- 172.21.15.113:0/2342475547 >> v1:172.21.15.124:6789/0 conn(0x7f5a180b2590 legacy=0x7f5a18071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.474+0000 7f5a1d2ef700 1 -- 172.21.15.113:0/2342475547 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18161d90 con 0x7f5a180b3900 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1d2ef700 1 --2- 172.21.15.113:0/2342475547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b3900 0x7f5a180b2050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a140098f0 tx=0x7f5a14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48368020c352fb90 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1caee700 1 -- 172.21.15.113:0/2342475547 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14004230 con 0x7f5a180b3900 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1caee700 1 -- 172.21.15.113:0/2342475547 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a14004390 con 0x7f5a180b3900 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1caee700 1 -- 172.21.15.113:0/2342475547 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140036a0 con 0x7f5a180b3900 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 -- 172.21.15.113:0/2342475547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b3900 msgr2=0x7f5a180b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/2342475547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b3900 0x7f5a180b2050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5a140098f0 tx=0x7f5a14009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 -- 172.21.15.113:0/2342475547 shutdown_connections 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/2342475547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a18140f20 0x7f5a180b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/2342475547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b3900 0x7f5a180b2050 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 -- 172.21.15.113:0/2342475547 >> 172.21.15.113:0/2342475547 conn(0x7f5a180743b0 msgr2=0x7f5a1806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 -- 172.21.15.113:0/2342475547 shutdown_connections 2024-01-16T13:52:44.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.475+0000 7f5a1fd54700 1 -- 172.21.15.113:0/2342475547 wait complete. 2024-01-16T13:52:44.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 Processor -- start 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 -- start start 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 0x7f5a180cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a180b3900 0x7f5a181e7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a18140f20 0x7f5a18145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181e7ac0 con 0x7f5a180b2590 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18145740 con 0x7f5a18140f20 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1fd54700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180ce040 con 0x7f5a180b3900 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1daf0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 0x7f5a180cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1daf0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 0x7f5a180cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56024/0 (socket says 172.21.15.113:56024) 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1daf0700 1 -- 172.21.15.113:0/4013364509 learned_addr learned my addr 172.21.15.113:0/4013364509 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.477+0000 7f5a1e2f1700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a18140f20 0x7f5a18145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1d2ef700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a180b3900 0x7f5a181e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1daf0700 1 -- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a180b3900 msgr2=0x7f5a181e7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1daf0700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a180b3900 0x7f5a181e7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1daf0700 1 -- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a18140f20 msgr2=0x7f5a18145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1daf0700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a18140f20 0x7f5a18145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1daf0700 1 -- 172.21.15.113:0/4013364509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14009550 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1d2ef700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a180b3900 0x7f5a181e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1daf0700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 0x7f5a180cda60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c7f0 tx=0x7f5a0800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08004220 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08004380 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a180ce320 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080036a0 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.478+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a180ce8e0 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.479+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18067f60 con 0x7f5a180b2590 2024-01-16T13:52:44.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.479+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a08003800 con 0x7f5a180b2590 2024-01-16T13:52:44.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.480+0000 7f5a0e7fc700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f59fc06b350 0x7f59fc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.480+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a08089720 con 0x7f5a180b2590 2024-01-16T13:52:44.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.480+0000 7f5a1d2ef700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f59fc06b350 0x7f59fc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.480+0000 7f5a1d2ef700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f59fc06b350 0x7f59fc06d810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5a180b3d40 tx=0x7f5a14008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.484+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a080545a0 con 0x7f5a180b2590 2024-01-16T13:52:44.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.680+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a180afbc0 con 0x7f5a180b2590 2024-01-16T13:52:44.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.681+0000 7f5a0e7fc700 1 -- 172.21.15.113:0/4013364509 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f5a08057bc0 con 0x7f5a180b2590 2024-01-16T13:52:44.664 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:44.665 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:44.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f59fc06b350 msgr2=0x7f59fc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f59fc06b350 0x7f59fc06d810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5a180b3d40 tx=0x7f5a14008030 comp rx=0 tx=0).stop 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 msgr2=0x7f5a180cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 0x7f5a180cda60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c7f0 tx=0x7f5a0800cb00 comp rx=0 tx=0).stop 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 shutdown_connections 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f59fc06b350 0x7f59fc06d810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a18140f20 0x7f5a18145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a180b2590 0x7f5a180cda60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 --2- 172.21.15.113:0/4013364509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a180b3900 0x7f5a181e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 >> 172.21.15.113:0/4013364509 conn(0x7f5a180743b0 msgr2=0x7f5a18061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 shutdown_connections 2024-01-16T13:52:44.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.683+0000 7f5a1fd54700 1 -- 172.21.15.113:0/4013364509 wait complete. 2024-01-16T13:52:44.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T13:52:02.303855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T13:52:03.253775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:44.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:44.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.821+0000 7f187c6cb700 1 Processor -- start 2024-01-16T13:52:44.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.821+0000 7f187c6cb700 1 -- start start 2024-01-16T13:52:44.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187c6cb700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187415f570 0x7f18740709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187c6cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18740d0520 0x7f18740afd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187c6cb700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18740af880 con 0x7f18740b4310 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187c6cb700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1874070f10 con 0x7f187415f570 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187c6cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740aec70 con 0x7f18740d0520 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187415f570 0x7f18740709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187415f570 0x7f18740709d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56030/0 (socket says 172.21.15.113:56030) 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 -- 172.21.15.113:0/885135751 learned_addr learned my addr 172.21.15.113:0/885135751 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187ac68700 1 --2- 172.21.15.113:0/885135751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18740d0520 0x7f18740afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 -- 172.21.15.113:0/885135751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18740d0520 msgr2=0x7f18740afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 --2- 172.21.15.113:0/885135751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18740d0520 0x7f18740afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 -- 172.21.15.113:0/885135751 >> v1:172.21.15.124:6789/0 conn(0x7f18740b4310 legacy=0x7f18740cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 -- 172.21.15.113:0/885135751 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18740aef50 con 0x7f187415f570 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.822+0000 7f187a467700 1 --2- 172.21.15.113:0/885135751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187415f570 0x7f18740709d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18700098f0 tx=0x7f1870009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69927eaefb2de01f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f1879465700 1 -- 172.21.15.113:0/885135751 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187000b8a0 con 0x7f187415f570 2024-01-16T13:52:44.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f1879465700 1 -- 172.21.15.113:0/885135751 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187000ba00 con 0x7f187415f570 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f1879465700 1 -- 172.21.15.113:0/885135751 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187000bd90 con 0x7f187415f570 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 -- 172.21.15.113:0/885135751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187415f570 msgr2=0x7f18740709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 --2- 172.21.15.113:0/885135751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187415f570 0x7f18740709d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18700098f0 tx=0x7f1870009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 -- 172.21.15.113:0/885135751 shutdown_connections 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 --2- 172.21.15.113:0/885135751 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18740d0520 0x7f18740afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 --2- 172.21.15.113:0/885135751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f187415f570 0x7f18740709d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 -- 172.21.15.113:0/885135751 >> 172.21.15.113:0/885135751 conn(0x7f18740b37f0 msgr2=0x7f18740b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 -- 172.21.15.113:0/885135751 shutdown_connections 2024-01-16T13:52:44.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.823+0000 7f187c6cb700 1 -- 172.21.15.113:0/885135751 wait complete. 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.824+0000 7f187c6cb700 1 Processor -- start 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.824+0000 7f187c6cb700 1 -- start start 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f187c6cb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18740b4310 0x7f18741f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f187c6cb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 0x7f187415b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f187c6cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187415f570 0x7f18740ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f187c6cb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187406aff0 con 0x7f18740d0520 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f187c6cb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740cd460 con 0x7f18740b4310 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f187c6cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18741f4ea0 con 0x7f187415f570 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f1879c66700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 0x7f187415b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f1879c66700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 0x7f187415b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56034/0 (socket says 172.21.15.113:56034) 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.825+0000 7f1879c66700 1 -- 172.21.15.113:0/1642061526 learned_addr learned my addr 172.21.15.113:0/1642061526 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f187a467700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18740b4310 0x7f18741f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f187ac68700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187415f570 0x7f18740ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f1879c66700 1 -- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187415f570 msgr2=0x7f18740ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f1879c66700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187415f570 0x7f18740ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f1879c66700 1 -- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18740b4310 msgr2=0x7f18741f4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f1879c66700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18740b4310 0x7f18741f4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f1879c66700 1 -- 172.21.15.113:0/1642061526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870009550 con 0x7f18740d0520 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f187a467700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18740b4310 0x7f18741f4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f1879c66700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 0x7f187415b1a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f186400c4e0 tx=0x7f186400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640052d0 con 0x7f18740d0520 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864005430 con 0x7f18740d0520 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18740cd180 con 0x7f18740d0520 2024-01-16T13:52:44.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640038c0 con 0x7f18740d0520 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.826+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18740cc8b0 con 0x7f18740d0520 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.827+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18740cc430 con 0x7f18740d0520 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.828+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1864003a20 con 0x7f18740d0520 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.828+0000 7f186affd700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185806b350 0x7f185806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.828+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1864089070 con 0x7f18740d0520 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.828+0000 7f187a467700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185806b350 0x7f185806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:44.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.828+0000 7f187a467700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185806b350 0x7f185806d810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1870014040 tx=0x7f1870008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:44.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:44.832+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18740cc430 con 0x7f18740d0520 2024-01-16T13:52:45.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.025+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f187406b130 con 0x7f18740d0520 2024-01-16T13:52:45.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.026+0000 7f186affd700 1 -- 172.21.15.113:0/1642061526 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f187406b130 con 0x7f18740d0520 2024-01-16T13:52:45.009 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:45.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185806b350 msgr2=0x7f185806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185806b350 0x7f185806d810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1870014040 tx=0x7f1870008030 comp rx=0 tx=0).stop 2024-01-16T13:52:45.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 msgr2=0x7f187415b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 0x7f187415b1a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f186400c4e0 tx=0x7f186400c8a0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 shutdown_connections 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f185806b350 0x7f185806d810 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f187415f570 0x7f18740ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18740b4310 0x7f18741f4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 --2- 172.21.15.113:0/1642061526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740d0520 0x7f187415b1a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 >> 172.21.15.113:0/1642061526 conn(0x7f18740b37f0 msgr2=0x7f1874070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 shutdown_connections 2024-01-16T13:52:45.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.028+0000 7f187c6cb700 1 -- 172.21.15.113:0/1642061526 wait complete. 2024-01-16T13:52:45.023 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T13:52:45.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.164+0000 7f40ee31f700 1 Processor -- start 2024-01-16T13:52:45.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.164+0000 7f40ee31f700 1 -- start start 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ee31f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f40e80b5220 0x7f40e80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ee31f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40e8073f90 0x7f40e806e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ee31f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e80b42a0 con 0x7f40e815e280 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ee31f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e80af900 con 0x7f40e8073f90 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ee31f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e8074d30 con 0x7f40e80b5220 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ec8bc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f40e8073f90 0x7f40e806e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40e7fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f40e815e280 0x7f40e80b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53424/0 (socket says 172.21.15.113:53424) 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40e7fff700 1 -- 172.21.15.113:0/3257123333 learned_addr learned my addr 172.21.15.113:0/3257123333 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40e77fe700 1 --2- 172.21.15.113:0/3257123333 >> v2:172.21.15.124:3300/0 conn(0x7f40e80b5220 0x7f40e80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ec8bc700 1 -- 172.21.15.113:0/3257123333 >> v2:172.21.15.124:3300/0 conn(0x7f40e80b5220 msgr2=0x7f40e80b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ec8bc700 1 --2- 172.21.15.113:0/3257123333 >> v2:172.21.15.124:3300/0 conn(0x7f40e80b5220 0x7f40e80b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ec8bc700 1 -- 172.21.15.113:0/3257123333 >> v1:172.21.15.113:6789/0 conn(0x7f40e815e280 legacy=0x7f40e80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ec8bc700 1 -- 172.21.15.113:0/3257123333 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e8074490 con 0x7f40e8073f90 2024-01-16T13:52:45.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40e6ffd700 1 -- 172.21.15.113:0/3257123333 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989254883 0 0) 0x7f40dc001110 con 0x7f40e815e280 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.165+0000 7f40ec8bc700 1 --2- 172.21.15.113:0/3257123333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e8073f90 0x7f40e806e220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f40e80ae210 tx=0x7f40e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0ed202e5c40431 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40e6ffd700 1 -- 172.21.15.113:0/3257123333 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e000e070 con 0x7f40e8073f90 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40e6ffd700 1 -- 172.21.15.113:0/3257123333 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e000bbd0 con 0x7f40e8073f90 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40e6ffd700 1 -- 172.21.15.113:0/3257123333 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0012430 con 0x7f40e8073f90 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 -- 172.21.15.113:0/3257123333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e8073f90 msgr2=0x7f40e806e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 --2- 172.21.15.113:0/3257123333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e8073f90 0x7f40e806e220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f40e80ae210 tx=0x7f40e000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 -- 172.21.15.113:0/3257123333 shutdown_connections 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 --2- 172.21.15.113:0/3257123333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e8073f90 0x7f40e806e220 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 --2- 172.21.15.113:0/3257123333 >> v2:172.21.15.124:3300/0 conn(0x7f40e80b5220 0x7f40e80b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 -- 172.21.15.113:0/3257123333 >> 172.21.15.113:0/3257123333 conn(0x7f40e8072b20 msgr2=0x7f40e806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 -- 172.21.15.113:0/3257123333 shutdown_connections 2024-01-16T13:52:45.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.166+0000 7f40ee31f700 1 -- 172.21.15.113:0/3257123333 wait complete. 2024-01-16T13:52:45.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.167+0000 7f40ee31f700 1 Processor -- start 2024-01-16T13:52:45.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 -- start start 2024-01-16T13:52:45.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f40e8073f90 0x7f40e8157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f40e80b5220 0x7f40e80c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 0x7f40e80c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e80c72a0 con 0x7f40e81f7160 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e80c6690 con 0x7f40e8073f90 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ee31f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e80c6970 con 0x7f40e80b5220 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ec8bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 0x7f40e80c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ec8bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 0x7f40e80c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56050/0 (socket says 172.21.15.113:56050) 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.168+0000 7f40ec8bc700 1 -- 172.21.15.113:0/2515107620 learned_addr learned my addr 172.21.15.113:0/2515107620 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40e7fff700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f40e8073f90 0x7f40e8157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40e77fe700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f40e80b5220 0x7f40e80c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ec8bc700 1 -- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f40e80b5220 msgr2=0x7f40e80c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ec8bc700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f40e80b5220 0x7f40e80c5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ec8bc700 1 -- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f40e8073f90 msgr2=0x7f40e8157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ec8bc700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f40e8073f90 0x7f40e8157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ec8bc700 1 -- 172.21.15.113:0/2515107620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e0009a70 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40e7fff700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f40e8073f90 0x7f40e8157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ec8bc700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 0x7f40e80c8250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f40e0013000 tx=0x7f40e0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0012430 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e0020070 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e80c6c50 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0004280 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.169+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e8157a70 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.170+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e8069020 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.170+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f40e00043e0 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.171+0000 7f40e4ff9700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f40d406b350 0x7f40d406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.171+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40e0022070 con 0x7f40e81f7160 2024-01-16T13:52:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.171+0000 7f40e7fff700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f40d406b350 0x7f40d406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.171+0000 7f40e7fff700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f40d406b350 0x7f40d406d810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f40dc007fd0 tx=0x7f40dc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.175+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40e00569b0 con 0x7f40e81f7160 2024-01-16T13:52:45.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.366+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40e806f8e0 con 0x7f40e81f7160 2024-01-16T13:52:45.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.366+0000 7f40e4ff9700 1 -- 172.21.15.113:0/2515107620 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40e0059fd0 con 0x7f40e81f7160 2024-01-16T13:52:45.349 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-01-16T13:52:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f40d406b350 msgr2=0x7f40d406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f40d406b350 0x7f40d406d810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f40dc007fd0 tx=0x7f40dc009040 comp rx=0 tx=0).stop 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 msgr2=0x7f40e80c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 0x7f40e80c8250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f40e0013000 tx=0x7f40e0004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 shutdown_connections 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f40d406b350 0x7f40d406d810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40e81f7160 0x7f40e80c8250 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f40e8073f90 0x7f40e8157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 --2- 172.21.15.113:0/2515107620 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f40e80b5220 0x7f40e80c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 >> 172.21.15.113:0/2515107620 conn(0x7f40e8072b20 msgr2=0x7f40e806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 shutdown_connections 2024-01-16T13:52:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.368+0000 7f40ee31f700 1 -- 172.21.15.113:0/2515107620 wait complete. 2024-01-16T13:52:45.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T13:52:45.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.504+0000 7f8413d31700 1 Processor -- start 2024-01-16T13:52:45.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f8413d31700 1 -- start start 2024-01-16T13:52:45.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f8413d31700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 0x7f840c0b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f8413d31700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c15a850 con 0x7f840c15df00 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f8413d31700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c06d470 con 0x7f840c15a470 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f8413d31700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0b29d0 con 0x7f840c0b1fe0 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f84112cc700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f840c15df00 0x7f840c0b33e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53434/0 (socket says 172.21.15.113:53434) 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.505+0000 7f84112cc700 1 -- 172.21.15.113:0/2709739176 learned_addr learned my addr 172.21.15.113:0/2709739176 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f84122ce700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 0x7f840c0b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165173626 0 0) 0x7f840c15a850 con 0x7f840c15df00 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f80046c0 con 0x7f840c15df00 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987029809 0 0) 0x7f840c06d470 con 0x7f840c15a470 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c15a850 con 0x7f840c15a470 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3025129526 0 0) 0x7f83f80046c0 con 0x7f840c15df00 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 msgr2=0x7f840c0b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 0x7f840c0b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 >> v1:172.21.15.124:6789/0 conn(0x7f840c15a470 legacy=0x7f840c0ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c15e430 con 0x7f840c15df00 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f8408001670 con 0x7f840c15df00 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 >> v1:172.21.15.113:6789/0 conn(0x7f840c15df00 legacy=0x7f840c0b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83f8004cd0 0x7f83f80050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83f8005960 0x7f83f8005de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 0x7f83f8006320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c0b29d0 con 0x7f83f8005960 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f800cc00 con 0x7f83f8004cd0 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f800cdc0 con 0x7f840c0b1fe0 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f84112cc700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83f8005960 0x7f83f8005de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f8411acd700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83f8004cd0 0x7f83f80050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f84122ce700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 0x7f83f8006320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f84112cc700 1 -- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 msgr2=0x7f83f8006320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.506+0000 7f84112cc700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f840c0b1fe0 0x7f83f8006320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f84112cc700 1 -- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83f8004cd0 msgr2=0x7f83f80050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f84112cc700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83f8004cd0 0x7f83f80050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f84112cc700 1 -- 172.21.15.113:0/2709739176 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f80046c0 con 0x7f83f8005960 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f84112cc700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83f8005960 0x7f83f8005de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8408001eb0 tx=0x7f83f8006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d486f40e819f7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840800cd20 con 0x7f83f8005960 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840800ce80 con 0x7f83f8005960 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8410acb700 1 -- 172.21.15.113:0/2709739176 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408005b60 con 0x7f83f8005960 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8413d31700 1 -- 172.21.15.113:0/2709739176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83f8005960 msgr2=0x7f83f8005de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8413d31700 1 --2- 172.21.15.113:0/2709739176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83f8005960 0x7f83f8005de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8408001eb0 tx=0x7f83f8006990 comp rx=0 tx=0).stop 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f84122ce700 1 -- 172.21.15.113:0/2709739176 reap_dead start 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8413d31700 1 -- 172.21.15.113:0/2709739176 shutdown_connections 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8413d31700 1 -- 172.21.15.113:0/2709739176 >> 172.21.15.113:0/2709739176 conn(0x7f840c074170 msgr2=0x7f840c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8413d31700 1 -- 172.21.15.113:0/2709739176 shutdown_connections 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.507+0000 7f8413d31700 1 -- 172.21.15.113:0/2709739176 wait complete. 2024-01-16T13:52:45.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 Processor -- start 2024-01-16T13:52:45.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 -- start start 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83f8005960 0x7f840c1ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 0x7f840c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f840c153120 0x7f840c0ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1535d0 con 0x7f840c142310 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c13f5c0 con 0x7f840c153120 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f8413d31700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1ee670 con 0x7f83f8005960 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f84112cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 0x7f840c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f84112cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 0x7f840c152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56080/0 (socket says 172.21.15.113:56080) 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.509+0000 7f84112cc700 1 -- 172.21.15.113:0/120900537 learned_addr learned my addr 172.21.15.113:0/120900537 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84122ce700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f840c153120 0x7f840c0ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f8411acd700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83f8005960 0x7f840c1ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84112cc700 1 -- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83f8005960 msgr2=0x7f840c1ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84112cc700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83f8005960 0x7f840c1ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84112cc700 1 -- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f840c153120 msgr2=0x7f840c0ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84112cc700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f840c153120 0x7f840c0ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84112cc700 1 -- 172.21.15.113:0/120900537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c1f4510 con 0x7f840c142310 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84112cc700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 0x7f840c152be0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f840800c7d0 tx=0x7f8408001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080072d0 con 0x7f840c142310 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8408007430 con 0x7f840c142310 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080077e0 con 0x7f840c142310 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840c1f47f0 con 0x7f840c142310 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.510+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840c1f4db0 con 0x7f840c142310 2024-01-16T13:52:45.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.511+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840c153530 con 0x7f840c142310 2024-01-16T13:52:45.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.512+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f840801e430 con 0x7f840c142310 2024-01-16T13:52:45.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.512+0000 7f84027fc700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f83f006b350 0x7f83f006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.512+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f840808a8c0 con 0x7f840c142310 2024-01-16T13:52:45.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.512+0000 7f8411acd700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f83f006b350 0x7f83f006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.512+0000 7f8411acd700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f83f006b350 0x7f83f006d810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f840c0af490 tx=0x7f83fc009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.516+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84080557c0 con 0x7f840c142310 2024-01-16T13:52:45.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.708+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f840c069020 con 0x7f840c142310 2024-01-16T13:52:45.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.709+0000 7f84027fc700 1 -- 172.21.15.113:0/120900537 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8408058de0 con 0x7f840c142310 2024-01-16T13:52:45.692 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 64 2024-01-16T13:52:45.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f83f006b350 msgr2=0x7f83f006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f83f006b350 0x7f83f006d810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f840c0af490 tx=0x7f83fc009040 comp rx=0 tx=0).stop 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 msgr2=0x7f840c152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 0x7f840c152be0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f840800c7d0 tx=0x7f8408001980 comp rx=0 tx=0).stop 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 shutdown_connections 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f83f006b350 0x7f83f006d810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f840c153120 0x7f840c0ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83f8005960 0x7f840c1ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 --2- 172.21.15.113:0/120900537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f840c142310 0x7f840c152be0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 >> 172.21.15.113:0/120900537 conn(0x7f840c074170 msgr2=0x7f840c0b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 shutdown_connections 2024-01-16T13:52:45.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.711+0000 7f8413d31700 1 -- 172.21.15.113:0/120900537 wait complete. 2024-01-16T13:52:45.706 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:45.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.845+0000 7f8d9a077700 1 Processor -- start 2024-01-16T13:52:45.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.845+0000 7f8d9a077700 1 -- start start 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d9a077700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 0x7f8d940b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d9a077700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8d9406bfc0 0x7f8d94073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d9a077700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d940b5ab0 con 0x7f8d940744d0 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d9a077700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9415c400 con 0x7f8d9406bfc0 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d9a077700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94074940 con 0x7f8d9415c020 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d93fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f8d940744d0 0x7f8d94071b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53450/0 (socket says 172.21.15.113:53450) 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d93fff700 1 -- 172.21.15.113:0/1664036594 learned_addr learned my addr 172.21.15.113:0/1664036594 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:45.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d937fe700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 0x7f8d940b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d92ffd700 1 --2- 172.21.15.113:0/1664036594 >> v2:172.21.15.124:3300/0 conn(0x7f8d9406bfc0 0x7f8d94073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154165830 0 0) 0x7f8d940b5ab0 con 0x7f8d940744d0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c0035b0 con 0x7f8d940744d0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141464752 0 0) 0x7f8d7c0035b0 con 0x7f8d940744d0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.846+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 msgr2=0x7f8d940b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 0x7f8d940b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 >> v2:172.21.15.124:3300/0 conn(0x7f8d9406bfc0 msgr2=0x7f8d94073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 --2- 172.21.15.113:0/1664036594 >> v2:172.21.15.124:3300/0 conn(0x7f8d9406bfc0 0x7f8d94073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9406c440 con 0x7f8d940744d0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f8d8c0019f0 con 0x7f8d940744d0 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 >> v1:172.21.15.113:6789/0 conn(0x7f8d940744d0 legacy=0x7f8d94071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 0x7f8d7c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d7c004d90 0x7f8d7c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9415c400 con 0x7f8d7c003a00 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94074940 con 0x7f8d7c004d90 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00ba10 con 0x7f8d9415c020 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d92ffd700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 0x7f8d7c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d93fff700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d7c004d90 0x7f8d7c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 -- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 msgr2=0x7f8d7c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d9415c020 0x7f8d7c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 -- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d7c004d90 msgr2=0x7f8d7c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d7c004d90 0x7f8d7c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 -- 172.21.15.113:0/1664036594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0035b0 con 0x7f8d7c003a00 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d93fff700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d7c004d90 0x7f8d7c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d937fe700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009b60 tx=0x7f8d7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13f712933433f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800e6c0 con 0x7f8d7c003a00 2024-01-16T13:52:45.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8800e820 con 0x7f8d7c003a00 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.847+0000 7f8d927fc700 1 -- 172.21.15.113:0/1664036594 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800eb90 con 0x7f8d7c003a00 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d9a077700 1 -- 172.21.15.113:0/1664036594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 msgr2=0x7f8d7c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d9a077700 1 --2- 172.21.15.113:0/1664036594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009b60 tx=0x7f8d7c005880 comp rx=0 tx=0).stop 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d93fff700 1 -- 172.21.15.113:0/1664036594 reap_dead start 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d9a077700 1 -- 172.21.15.113:0/1664036594 shutdown_connections 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d9a077700 1 -- 172.21.15.113:0/1664036594 >> 172.21.15.113:0/1664036594 conn(0x7f8d940b3190 msgr2=0x7f8d940d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d9a077700 1 -- 172.21.15.113:0/1664036594 shutdown_connections 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.848+0000 7f8d9a077700 1 -- 172.21.15.113:0/1664036594 wait complete. 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.849+0000 7f8d9a077700 1 Processor -- start 2024-01-16T13:52:45.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.849+0000 7f8d9a077700 1 -- start start 2024-01-16T13:52:45.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d9a077700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d940cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d9a077700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d94147140 0x7f8d941475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d9a077700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9415e410 0x7f8d9415e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d9a077700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9415ee00 con 0x7f8d7c003a00 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d9a077700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94147b00 con 0x7f8d9415e410 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d9a077700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9415d4c0 con 0x7f8d94147140 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d940cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d940cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56110/0 (socket says 172.21.15.113:56110) 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 -- 172.21.15.113:0/472583332 learned_addr learned my addr 172.21.15.113:0/472583332 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 -- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d94147140 msgr2=0x7f8d941475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d94147140 0x7f8d941475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 -- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9415e410 msgr2=0x7f8d9415e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d93fff700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9415e410 0x7f8d9415e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9415e410 0x7f8d9415e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 -- 172.21.15.113:0/472583332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9415c8b0 con 0x7f8d7c003a00 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.850+0000 7f8d937fe700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d940cec60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800e4b0 tx=0x7f8d880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.851+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800e6c0 con 0x7f8d7c003a00 2024-01-16T13:52:45.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.851+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d88003ad0 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.851+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9415cb90 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.851+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88003e80 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.851+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9415d150 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.852+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d94067f60 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.852+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8d88013470 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.852+0000 7f8d7bfff700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f8d8006b350 0x7f8d8006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.853+0000 7f8d92ffd700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f8d8006b350 0x7f8d8006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.853+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d8808b380 con 0x7f8d7c003a00 2024-01-16T13:52:45.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.853+0000 7f8d92ffd700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f8d8006b350 0x7f8d8006d810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d84006f10 tx=0x7f8d84006e80 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:45.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:45.857+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d88056280 con 0x7f8d7c003a00 2024-01-16T13:52:46.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.090+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d940b2610 con 0x7f8d7c003a00 2024-01-16T13:52:46.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.090+0000 7f8d7bfff700 1 -- 172.21.15.113:0/472583332 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f8d880598a0 con 0x7f8d7c003a00 2024-01-16T13:52:46.073 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:46.073 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:46.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f8d8006b350 msgr2=0x7f8d8006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f8d8006b350 0x7f8d8006d810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d84006f10 tx=0x7f8d84006e80 comp rx=0 tx=0).stop 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 msgr2=0x7f8d940cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d940cec60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800e4b0 tx=0x7f8d880047c0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 shutdown_connections 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f8d8006b350 0x7f8d8006d810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9415e410 0x7f8d9415e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d7c003a00 0x7f8d940cec60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 --2- 172.21.15.113:0/472583332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8d94147140 0x7f8d941475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 >> 172.21.15.113:0/472583332 conn(0x7f8d940b3190 msgr2=0x7f8d940b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.092+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 shutdown_connections 2024-01-16T13:52:46.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.093+0000 7f8d9a077700 1 -- 172.21.15.113:0/472583332 wait complete. 2024-01-16T13:52:46.077 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:46.088 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:46.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.225+0000 7fe4a1d06700 1 Processor -- start 2024-01-16T13:52:46.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.225+0000 7fe4a1d06700 1 -- start start 2024-01-16T13:52:46.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe4a1d06700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe49c06ee50 0x7fe49c074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe4a1d06700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe49c15c140 0x7fe49c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe4a1d06700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c074e90 con 0x7fe49c073f90 2024-01-16T13:52:46.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe4a1d06700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c0748b0 con 0x7fe49c15c140 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe4a1d06700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c15c700 con 0x7fe49c06ee50 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49bfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe49c15c140 0x7fe49c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49b7fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe49c073f90 0x7fe49c0ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53454/0 (socket says 172.21.15.113:53454) 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49b7fe700 1 -- 172.21.15.113:0/1753745873 learned_addr learned my addr 172.21.15.113:0/1753745873 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49affd700 1 --2- 172.21.15.113:0/1753745873 >> v2:172.21.15.124:3300/0 conn(0x7fe49c06ee50 0x7fe49c074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49bfff700 1 -- 172.21.15.113:0/1753745873 >> v2:172.21.15.124:3300/0 conn(0x7fe49c06ee50 msgr2=0x7fe49c074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49bfff700 1 --2- 172.21.15.113:0/1753745873 >> v2:172.21.15.124:3300/0 conn(0x7fe49c06ee50 0x7fe49c074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49bfff700 1 -- 172.21.15.113:0/1753745873 >> v1:172.21.15.113:6789/0 conn(0x7fe49c073f90 legacy=0x7fe49c0ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49a7fc700 1 -- 172.21.15.113:0/1753745873 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321250931 0 0) 0x7fe49c074e90 con 0x7fe49c073f90 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49bfff700 1 -- 172.21.15.113:0/1753745873 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c15c9e0 con 0x7fe49c15c140 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49bfff700 1 --2- 172.21.15.113:0/1753745873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c15c140 0x7fe49c0cf290 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe494009d90 tx=0x7fe49400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1928177ccd64af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49a7fc700 1 -- 172.21.15.113:0/1753745873 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49400e070 con 0x7fe49c15c140 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49a7fc700 1 -- 172.21.15.113:0/1753745873 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49400bae0 con 0x7fe49c15c140 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.226+0000 7fe49a7fc700 1 -- 172.21.15.113:0/1753745873 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49400be90 con 0x7fe49c15c140 2024-01-16T13:52:46.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 -- 172.21.15.113:0/1753745873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c15c140 msgr2=0x7fe49c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/1753745873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c15c140 0x7fe49c0cf290 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe494009d90 tx=0x7fe49400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 -- 172.21.15.113:0/1753745873 shutdown_connections 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/1753745873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c15c140 0x7fe49c0cf290 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/1753745873 >> v2:172.21.15.124:3300/0 conn(0x7fe49c06ee50 0x7fe49c074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 -- 172.21.15.113:0/1753745873 >> 172.21.15.113:0/1753745873 conn(0x7fe49c0cd570 msgr2=0x7fe49c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 -- 172.21.15.113:0/1753745873 shutdown_connections 2024-01-16T13:52:46.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.227+0000 7fe4a1d06700 1 -- 172.21.15.113:0/1753745873 wait complete. 2024-01-16T13:52:46.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.228+0000 7fe4a1d06700 1 Processor -- start 2024-01-16T13:52:46.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.228+0000 7fe4a1d06700 1 -- start start 2024-01-16T13:52:46.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe4a1d06700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 0x7fe49c13d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe4a1d06700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe49c073f90 0x7fe49c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe4a1d06700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe49c15c140 0x7fe49c15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe4a1d06700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c15ec10 con 0x7fe49c06ee50 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe4a1d06700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c15eef0 con 0x7fe49c073f90 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe4a1d06700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c15f1d0 con 0x7fe49c15c140 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 0x7fe49c13d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 0x7fe49c13d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56116/0 (socket says 172.21.15.113:56116) 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49b7fe700 1 -- 172.21.15.113:0/3518325998 learned_addr learned my addr 172.21.15.113:0/3518325998 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49bfff700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe49c15c140 0x7fe49c15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49affd700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe49c073f90 0x7fe49c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49b7fe700 1 -- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe49c15c140 msgr2=0x7fe49c15a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.229+0000 7fe49b7fe700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe49c15c140 0x7fe49c15a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe49b7fe700 1 -- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe49c073f90 msgr2=0x7fe49c15fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe49b7fe700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe49c073f90 0x7fe49c15fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe49b7fe700 1 -- 172.21.15.113:0/3518325998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494009a70 con 0x7fe49c06ee50 2024-01-16T13:52:46.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe49b7fe700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 0x7fe49c13d470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0011e0 tx=0x7fe48c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c004c80 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c004e00 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c007310 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c15f4b0 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.230+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c13d9b0 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.231+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c15f9d0 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.231+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe48c007470 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.232+0000 7fe483fff700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe48806b350 0x7fe48806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.232+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe48c089940 con 0x7fe49c06ee50 2024-01-16T13:52:46.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.232+0000 7fe49affd700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe48806b350 0x7fe48806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.232+0000 7fe49affd700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe48806b350 0x7fe48806d810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe49c140db0 tx=0x7fe490008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.236+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe49c15f9d0 con 0x7fe49c06ee50 2024-01-16T13:52:46.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.427+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe49c15f9d0 con 0x7fe49c06ee50 2024-01-16T13:52:46.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.428+0000 7fe483fff700 1 -- 172.21.15.113:0/3518325998 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fe49c15f9d0 con 0x7fe49c06ee50 2024-01-16T13:52:46.411 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:46.412 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe48806b350 msgr2=0x7fe48806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe48806b350 0x7fe48806d810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe49c140db0 tx=0x7fe490008040 comp rx=0 tx=0).stop 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 msgr2=0x7fe49c13d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 0x7fe49c13d470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0011e0 tx=0x7fe48c00cae0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 shutdown_connections 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe48806b350 0x7fe48806d810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe49c15c140 0x7fe49c15a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c06ee50 0x7fe49c13d470 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 --2- 172.21.15.113:0/3518325998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe49c073f90 0x7fe49c15fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.430+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 >> 172.21.15.113:0/3518325998 conn(0x7fe49c0cd570 msgr2=0x7fe49c06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:46.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.431+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 shutdown_connections 2024-01-16T13:52:46.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.431+0000 7fe4a1d06700 1 -- 172.21.15.113:0/3518325998 wait complete. 2024-01-16T13:52:46.426 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T13:52:46.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:46.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.565+0000 7f482b5a2700 1 Processor -- start 2024-01-16T13:52:46.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.565+0000 7f482b5a2700 1 -- start start 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.566+0000 7f482b5a2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f482415e080 0x7f48240cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.566+0000 7f482b5a2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240d0150 con 0x7f48240d22f0 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.566+0000 7f482b5a2700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240cee50 con 0x7f48240ce4d0 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.566+0000 7f482b5a2700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482415e460 con 0x7f482415e080 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.566+0000 7f4829b3f700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f48240d22f0 0x7f48240d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53468/0 (socket says 172.21.15.113:53468) 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.566+0000 7f4829b3f700 1 -- 172.21.15.113:0/660403268 learned_addr learned my addr 172.21.15.113:0/660403268 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f482933e700 1 --2- 172.21.15.113:0/660403268 >> v2:172.21.15.124:3300/0 conn(0x7f482415e080 0x7f48240cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f481bfff700 1 -- 172.21.15.113:0/660403268 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428439782 0 0) 0x7f48240d0150 con 0x7f48240d22f0 2024-01-16T13:52:46.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f481bfff700 1 -- 172.21.15.113:0/660403268 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48100035b0 con 0x7f48240d22f0 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f481bfff700 1 -- 172.21.15.113:0/660403268 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245962082 0 0) 0x7f48100035b0 con 0x7f48240d22f0 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f482933e700 1 -- 172.21.15.113:0/660403268 >> v1:172.21.15.124:6789/0 conn(0x7f48240ce4d0 legacy=0x7f48240d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f482933e700 1 -- 172.21.15.113:0/660403268 >> v1:172.21.15.113:6789/0 conn(0x7f48240d22f0 legacy=0x7f48240d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f482933e700 1 -- 172.21.15.113:0/660403268 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48240cf130 con 0x7f482415e080 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f482933e700 1 --2- 172.21.15.113:0/660403268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f482415e080 0x7f48240cc2a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4814009840 tx=0x7f4814009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70e07012fa3bda85 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f481bfff700 1 -- 172.21.15.113:0/660403268 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c0019f0 con 0x7f482415e080 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f481bfff700 1 -- 172.21.15.113:0/660403268 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814004030 con 0x7f482415e080 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.567+0000 7f481bfff700 1 -- 172.21.15.113:0/660403268 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48140043a0 con 0x7f482415e080 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 -- 172.21.15.113:0/660403268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f482415e080 msgr2=0x7f48240cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 --2- 172.21.15.113:0/660403268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f482415e080 0x7f48240cc2a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4814009840 tx=0x7f4814009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 -- 172.21.15.113:0/660403268 shutdown_connections 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 --2- 172.21.15.113:0/660403268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f482415e080 0x7f48240cc2a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 -- 172.21.15.113:0/660403268 >> 172.21.15.113:0/660403268 conn(0x7f4824073130 msgr2=0x7f482409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:46.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 -- 172.21.15.113:0/660403268 shutdown_connections 2024-01-16T13:52:46.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.568+0000 7f482b5a2700 1 -- 172.21.15.113:0/660403268 wait complete. 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.569+0000 7f482b5a2700 1 Processor -- start 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.569+0000 7f482b5a2700 1 -- start start 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f482b5a2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48240ce4d0 0x7f48240c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f482b5a2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48240d22f0 0x7f482414ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f482b5a2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 0x7f48240cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f482b5a2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240c5fd0 con 0x7f482415e080 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f482b5a2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240c62b0 con 0x7f48240d22f0 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f482b5a2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48240c6590 con 0x7f48240ce4d0 2024-01-16T13:52:46.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 0x7f48240cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 0x7f48240cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56124/0 (socket says 172.21.15.113:56124) 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 -- 172.21.15.113:0/3831399317 learned_addr learned my addr 172.21.15.113:0/3831399317 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 -- 172.21.15.113:0/3831399317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48240ce4d0 msgr2=0x7f48240c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48240ce4d0 0x7f48240c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 -- 172.21.15.113:0/3831399317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48240d22f0 msgr2=0x7f482414ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48240d22f0 0x7f482414ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 -- 172.21.15.113:0/3831399317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48140094a0 con 0x7f482415e080 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4829b3f700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 0x7f48240cb060 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f481c001870 tx=0x7f481c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.570+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c00ad40 con 0x7f482415e080 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.571+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c0070d0 con 0x7f482415e080 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.571+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48240c6870 con 0x7f482415e080 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.571+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c00aea0 con 0x7f482415e080 2024-01-16T13:52:46.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.571+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48240c5cc0 con 0x7f482415e080 2024-01-16T13:52:46.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.572+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48240c6d90 con 0x7f482415e080 2024-01-16T13:52:46.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.572+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f481c007480 con 0x7f482415e080 2024-01-16T13:52:46.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.572+0000 7f4819ffb700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f480806b350 0x7f480806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.573+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f481c089c60 con 0x7f482415e080 2024-01-16T13:52:46.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.573+0000 7f482933e700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f480806b350 0x7f480806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.573+0000 7f482933e700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f480806b350 0x7f480806d810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4814009b90 tx=0x7f4814010000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.577+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48240c6d90 con 0x7f482415e080 2024-01-16T13:52:46.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.814+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48240c6d90 con 0x7f482415e080 2024-01-16T13:52:46.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.815+0000 7f4819ffb700 1 -- 172.21.15.113:0/3831399317 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f48240c6d90 con 0x7f482415e080 2024-01-16T13:52:46.798 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:46.798 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:46.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f480806b350 msgr2=0x7f480806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f480806b350 0x7f480806d810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4814009b90 tx=0x7f4814010000 comp rx=0 tx=0).stop 2024-01-16T13:52:46.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 msgr2=0x7f48240cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 0x7f48240cb060 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f481c001870 tx=0x7f481c00a340 comp rx=0 tx=0).stop 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 shutdown_connections 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f480806b350 0x7f480806d810 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f482415e080 0x7f48240cb060 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48240ce4d0 0x7f48240c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 --2- 172.21.15.113:0/3831399317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48240d22f0 0x7f482414ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 >> 172.21.15.113:0/3831399317 conn(0x7f4824073130 msgr2=0x7f48240b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 shutdown_connections 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.817+0000 7f482b5a2700 1 -- 172.21.15.113:0/3831399317 wait complete. 2024-01-16T13:52:46.801 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:46.812 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:52:46.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.950+0000 7efcadccc700 1 Processor -- start 2024-01-16T13:52:46.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.950+0000 7efcadccc700 1 -- start start 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efcadccc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca8074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efcadccc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efca815c140 0x7efca80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efcadccc700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca8074e90 con 0x7efca8073f90 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efcadccc700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca80748b0 con 0x7efca815c140 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efcadccc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca815c700 con 0x7efca806ee50 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efca7fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efca815c140 0x7efca80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efca77fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7efca8073f90 0x7efca80ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53474/0 (socket says 172.21.15.113:53474) 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.951+0000 7efca77fe700 1 -- 172.21.15.113:0/1688380624 learned_addr learned my addr 172.21.15.113:0/1688380624 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca6ffd700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca8074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca7fff700 1 -- 172.21.15.113:0/1688380624 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 msgr2=0x7efca8074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca7fff700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca8074370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca7fff700 1 -- 172.21.15.113:0/1688380624 >> v1:172.21.15.113:6789/0 conn(0x7efca8073f90 legacy=0x7efca80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca7fff700 1 -- 172.21.15.113:0/1688380624 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca815c9e0 con 0x7efca815c140 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca67fc700 1 -- 172.21.15.113:0/1688380624 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45142380 0 0) 0x7efca8074e90 con 0x7efca8073f90 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca6ffd700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca8074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca7fff700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca80cf290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efca0009d90 tx=0x7efca000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e517b8d76006e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca67fc700 1 -- 172.21.15.113:0/1688380624 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca000e070 con 0x7efca815c140 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca67fc700 1 -- 172.21.15.113:0/1688380624 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca000bbd0 con 0x7efca815c140 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efca67fc700 1 -- 172.21.15.113:0/1688380624 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0012430 con 0x7efca815c140 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efcadccc700 1 -- 172.21.15.113:0/1688380624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 msgr2=0x7efca80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efcadccc700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca80cf290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efca0009d90 tx=0x7efca000b330 comp rx=0 tx=0).stop 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efcadccc700 1 -- 172.21.15.113:0/1688380624 shutdown_connections 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efcadccc700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca80cf290 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efcadccc700 1 --2- 172.21.15.113:0/1688380624 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca8074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.952+0000 7efcadccc700 1 -- 172.21.15.113:0/1688380624 >> 172.21.15.113:0/1688380624 conn(0x7efca80cd570 msgr2=0x7efca80cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.953+0000 7efcadccc700 1 -- 172.21.15.113:0/1688380624 shutdown_connections 2024-01-16T13:52:46.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.953+0000 7efcadccc700 1 -- 172.21.15.113:0/1688380624 wait complete. 2024-01-16T13:52:46.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.954+0000 7efcadccc700 1 Processor -- start 2024-01-16T13:52:46.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.954+0000 7efcadccc700 1 -- start start 2024-01-16T13:52:46.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efca8073f90 0x7efca80ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca814f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80cc2c0 con 0x7efca815c140 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca814f8c0 con 0x7efca8073f90 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca814fba0 con 0x7efca806ee50 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca814f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca814f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56148/0 (socket says 172.21.15.113:56148) 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 -- 172.21.15.113:0/1732797563 learned_addr learned my addr 172.21.15.113:0/1732797563 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca6ffd700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efca8073f90 0x7efca80ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca77fe700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 -- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 msgr2=0x7efca815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 -- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efca8073f90 msgr2=0x7efca80ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efca8073f90 0x7efca80ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 -- 172.21.15.113:0/1732797563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0009a70 con 0x7efca815c140 2024-01-16T13:52:46.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca6ffd700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efca8073f90 0x7efca80ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efca7fff700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca814f230 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efca8073510 tx=0x7efca0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0003ea0 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0020070 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca814fe80 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0004280 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.955+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca814c660 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.956+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca81503a0 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.957+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efca00043e0 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.957+0000 7efc8ffff700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7efc9406b350 0x7efc9406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.957+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efca0022070 con 0x7efca815c140 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.957+0000 7efca77fe700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7efc9406b350 0x7efc9406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:46.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.957+0000 7efca77fe700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7efc9406b350 0x7efc9406d810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc9c000f60 tx=0x7efc9c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:46.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:46.962+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efca0056920 con 0x7efca815c140 2024-01-16T13:52:47.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.193+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efca806c810 con 0x7efca815c140 2024-01-16T13:52:47.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.194+0000 7efc8ffff700 1 -- 172.21.15.113:0/1732797563 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7efca002c090 con 0x7efca815c140 2024-01-16T13:52:47.177 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:47.177 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/158120732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":158120732},{"type":"v1","addr":"172.21.15.124:6835","nonce":158120732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2003877747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2003877747},{"type":"v1","addr":"172.21.15.113:6836","nonce":2003877747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1245807737","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":1245807737},{"type":"v1","addr":"172.21.15.113:6837","nonce":1245807737}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:04.292443+0000","modified":"2024-01-16T13:52:07.316433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6833/4249021810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":4249021810},{"type":"v1","addr":"172.21.15.124:6833","nonce":4249021810}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7efc9406b350 msgr2=0x7efc9406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7efc9406b350 0x7efc9406d810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc9c000f60 tx=0x7efc9c009040 comp rx=0 tx=0).stop 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 msgr2=0x7efca814f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca814f230 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efca8073510 tx=0x7efca0004ad0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 shutdown_connections 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7efc9406b350 0x7efc9406d810 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efca815c140 0x7efca814f230 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efca806ee50 0x7efca815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 --2- 172.21.15.113:0/1732797563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efca8073f90 0x7efca80ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 >> 172.21.15.113:0/1732797563 conn(0x7efca80cd570 msgr2=0x7efca806bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:47.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 shutdown_connections 2024-01-16T13:52:47.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.196+0000 7efcadccc700 1 -- 172.21.15.113:0/1732797563 wait complete. 2024-01-16T13:52:47.180 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 8 2024-01-16T13:52:47.191 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:47.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.329+0000 7ff4c97c9700 1 Processor -- start 2024-01-16T13:52:47.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.329+0000 7ff4c97c9700 1 -- start start 2024-01-16T13:52:47.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c97c9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff4c415e850 0x7ff4c406c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c97c9700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c406dd80 con 0x7ff4c40afe70 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c97c9700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c415f880 con 0x7ff4c415bef0 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c97c9700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c415c2d0 con 0x7ff4c415e850 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c37fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff4c415e850 0x7ff4c406c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c27fc700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff4c40afe70 0x7ff4c41624d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53476/0 (socket says 172.21.15.113:53476) 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c37fe700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff4c415e850 0x7ff4c406c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56162/0 (socket says 172.21.15.113:56162) 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c27fc700 1 -- 172.21.15.113:0/691878244 learned_addr learned my addr 172.21.15.113:0/691878244 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c37fe700 1 -- 172.21.15.113:0/691878244 >> v1:172.21.15.124:6789/0 conn(0x7ff4c415bef0 legacy=0x7ff4c40b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c37fe700 1 -- 172.21.15.113:0/691878244 >> v1:172.21.15.113:6789/0 conn(0x7ff4c40afe70 legacy=0x7ff4c41624d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c37fe700 1 -- 172.21.15.113:0/691878244 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c415efe0 con 0x7ff4c415e850 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.330+0000 7ff4c1ffb700 1 -- 172.21.15.113:0/691878244 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380752506 0 0) 0x7ff4c406dd80 con 0x7ff4c40afe70 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c37fe700 1 --2- 172.21.15.113:0/691878244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415e850 0x7ff4c406c000 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009d90 tx=0x7ff4bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852192fb29ed41a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c1ffb700 1 -- 172.21.15.113:0/691878244 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00e070 con 0x7ff4c415e850 2024-01-16T13:52:47.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c1ffb700 1 -- 172.21.15.113:0/691878244 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc00bb90 con 0x7ff4c415e850 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c1ffb700 1 -- 172.21.15.113:0/691878244 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc003a20 con 0x7ff4c415e850 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 -- 172.21.15.113:0/691878244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415e850 msgr2=0x7ff4c406c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/691878244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415e850 0x7ff4c406c000 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009d90 tx=0x7ff4bc00b330 comp rx=0 tx=0).stop 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 -- 172.21.15.113:0/691878244 shutdown_connections 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/691878244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415e850 0x7ff4c406c000 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 -- 172.21.15.113:0/691878244 >> 172.21.15.113:0/691878244 conn(0x7ff4c4071ac0 msgr2=0x7ff4c4071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 -- 172.21.15.113:0/691878244 shutdown_connections 2024-01-16T13:52:47.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.331+0000 7ff4c97c9700 1 -- 172.21.15.113:0/691878244 wait complete. 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 Processor -- start 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 -- start start 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff4c40afe70 0x7ff4c41556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 0x7ff4c40ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4c415e850 0x7ff4c415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4155bf0 con 0x7ff4c415bef0 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c40ae0c0 con 0x7ff4c415e850 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.333+0000 7ff4c97c9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c40ae3a0 con 0x7ff4c40afe70 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 0x7ff4c40ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 0x7ff4c40ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56174/0 (socket says 172.21.15.113:56174) 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 -- 172.21.15.113:0/759759045 learned_addr learned my addr 172.21.15.113:0/759759045 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c2ffd700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff4c40afe70 0x7ff4c41556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c37fe700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4c415e850 0x7ff4c415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 -- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff4c40afe70 msgr2=0x7ff4c41556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff4c40afe70 0x7ff4c41556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 -- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4c415e850 msgr2=0x7ff4c415cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4c415e850 0x7ff4c415cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 -- 172.21.15.113:0/759759045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc009a70 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c37fe700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4c415e850 0x7ff4c415cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c27fc700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 0x7ff4c40ad870 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c550 tx=0x7ff4b400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4004c80 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4004e00 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c40ae680 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4007310 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.334+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c40aec40 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.335+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c40aee60 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.336+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff4b4007470 con 0x7ff4c415bef0 2024-01-16T13:52:47.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.336+0000 7ff4ab7fe700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff4b006b350 0x7ff4b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.336+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4b40897e0 con 0x7ff4c415bef0 2024-01-16T13:52:47.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.336+0000 7ff4c2ffd700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff4b006b350 0x7ff4b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.336+0000 7ff4c2ffd700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff4b006b350 0x7ff4b006d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4162260 tx=0x7ff4b80090e0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:47.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.340+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4b4054650 con 0x7ff4c415bef0 2024-01-16T13:52:47.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.532+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c4067f60 con 0x7ff4c415bef0 2024-01-16T13:52:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.533+0000 7ff4ab7fe700 1 -- 172.21.15.113:0/759759045 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16541 (secure 0 0 0) 0x7ff4b4057c70 con 0x7ff4c415bef0 2024-01-16T13:52:47.516 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:47.516 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:05.899526+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:02.303855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:03.253775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff4b006b350 msgr2=0x7ff4b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff4b006b350 0x7ff4b006d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4162260 tx=0x7ff4b80090e0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 msgr2=0x7ff4c40ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 0x7ff4c40ad870 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c550 tx=0x7ff4b400c860 comp rx=0 tx=0).stop 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 shutdown_connections 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ff4b006b350 0x7ff4b006d810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4c415e850 0x7ff4c415cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff4c40afe70 0x7ff4c41556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 --2- 172.21.15.113:0/759759045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4c415bef0 0x7ff4c40ad870 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 >> 172.21.15.113:0/759759045 conn(0x7ff4c4071ac0 msgr2=0x7ff4c406db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:47.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 shutdown_connections 2024-01-16T13:52:47.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.535+0000 7ff4c97c9700 1 -- 172.21.15.113:0/759759045 wait complete. 2024-01-16T13:52:47.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-16T13:52:47.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.672+0000 7f78b2bea700 1 Processor -- start 2024-01-16T13:52:47.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.672+0000 7f78b2bea700 1 -- start start 2024-01-16T13:52:47.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78b2bea700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f78ac15bed0 0x7f78ac0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78b2bea700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78ac1603c0 0x7f78ac0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78b2bea700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac0d0770 con 0x7f78ac0d1a50 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78b2bea700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac15c2b0 con 0x7f78ac1603c0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78b2bea700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac0cd100 con 0x7f78ac15bed0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78ac1603c0 0x7f78ac0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f78ac1603c0 0x7f78ac0d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56184/0 (socket says 172.21.15.113:56184) 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 -- 172.21.15.113:0/1598590719 learned_addr learned my addr 172.21.15.113:0/1598590719 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78b0986700 1 --2- 172.21.15.113:0/1598590719 >> v2:172.21.15.124:3300/0 conn(0x7f78ac15bed0 0x7f78ac0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 -- 172.21.15.113:0/1598590719 >> v2:172.21.15.124:3300/0 conn(0x7f78ac15bed0 msgr2=0x7f78ac0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 --2- 172.21.15.113:0/1598590719 >> v2:172.21.15.124:3300/0 conn(0x7f78ac15bed0 0x7f78ac0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 -- 172.21.15.113:0/1598590719 >> v1:172.21.15.124:6789/0 conn(0x7f78ac0d1a50 legacy=0x7f78ac0d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 -- 172.21.15.113:0/1598590719 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0d09a0 con 0x7f78ac1603c0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.673+0000 7f78abfff700 1 --2- 172.21.15.113:0/1598590719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac1603c0 0x7f78ac0d0230 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f78ac09b2a0 tx=0x7f7898009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5562fb56553b48b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78ab7fe700 1 -- 172.21.15.113:0/1598590719 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898004230 con 0x7f78ac1603c0 2024-01-16T13:52:47.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78ab7fe700 1 -- 172.21.15.113:0/1598590719 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898004390 con 0x7f78ac1603c0 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78ab7fe700 1 -- 172.21.15.113:0/1598590719 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980036a0 con 0x7f78ac1603c0 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 -- 172.21.15.113:0/1598590719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac1603c0 msgr2=0x7f78ac0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 --2- 172.21.15.113:0/1598590719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac1603c0 0x7f78ac0d0230 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f78ac09b2a0 tx=0x7f7898009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 -- 172.21.15.113:0/1598590719 shutdown_connections 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 --2- 172.21.15.113:0/1598590719 >> v2:172.21.15.124:3300/0 conn(0x7f78ac15bed0 0x7f78ac0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 --2- 172.21.15.113:0/1598590719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac1603c0 0x7f78ac0d0230 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 -- 172.21.15.113:0/1598590719 >> 172.21.15.113:0/1598590719 conn(0x7f78ac073130 msgr2=0x7f78ac09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 -- 172.21.15.113:0/1598590719 shutdown_connections 2024-01-16T13:52:47.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.674+0000 7f78b2bea700 1 -- 172.21.15.113:0/1598590719 wait complete. 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.675+0000 7f78b2bea700 1 Processor -- start 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.675+0000 7f78b2bea700 1 -- start start 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b2bea700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 0x7f78ac15e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b2bea700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f78ac15bed0 0x7f78ac0cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b2bea700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78ac1603c0 0x7f78ac158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b2bea700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0cbc00 con 0x7f78ac0d1a50 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b2bea700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac15d1f0 con 0x7f78ac1603c0 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b2bea700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1594a0 con 0x7f78ac15bed0 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b0986700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 0x7f78ac15e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b0986700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 0x7f78ac15e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56186/0 (socket says 172.21.15.113:56186) 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.676+0000 7f78b0986700 1 -- 172.21.15.113:0/4195979034 learned_addr learned my addr 172.21.15.113:0/4195979034 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:47.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78abfff700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f78ac15bed0 0x7f78ac0cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b1187700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78ac1603c0 0x7f78ac158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b0986700 1 -- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f78ac15bed0 msgr2=0x7f78ac0cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b0986700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f78ac15bed0 0x7f78ac0cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b0986700 1 -- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78ac1603c0 msgr2=0x7f78ac158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b0986700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78ac1603c0 0x7f78ac158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b0986700 1 -- 172.21.15.113:0/4195979034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7898009550 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b0986700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 0x7f78ac15e5d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c7f0 tx=0x7f78a000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0011800 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0011960 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0011cf0 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac06b050 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.677+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac15f1c0 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.678+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac067f60 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.678+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f78a0018360 con 0x7f78ac0d1a50 2024-01-16T13:52:47.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.679+0000 7f78a97fa700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f789006b350 0x7f789006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:47.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.679+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a0089b20 con 0x7f78ac0d1a50 2024-01-16T13:52:47.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.679+0000 7f78abfff700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f789006b350 0x7f789006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:47.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.679+0000 7f78abfff700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f789006b350 0x7f789006d810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7898009130 tx=0x7f7898008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:47.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.683+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78a00549a0 con 0x7f78ac0d1a50 2024-01-16T13:52:47.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.916+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f78ac0b7bd0 con 0x7f78ac0d1a50 2024-01-16T13:52:47.900 INFO:tasks.ceph.mon.a.smithi113.stderr:2024-01-16T13:52:47.917+0000 7f9e78d0f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T13:52:47.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.920+0000 7f78a97fa700 1 -- 172.21.15.113:0/4195979034 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f78a0057fc0 con 0x7f78ac0d1a50 2024-01-16T13:52:47.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f789006b350 msgr2=0x7f789006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f789006b350 0x7f789006d810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7898009130 tx=0x7f7898008030 comp rx=0 tx=0).stop 2024-01-16T13:52:47.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 msgr2=0x7f78ac15e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:47.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 0x7f78ac15e5d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c7f0 tx=0x7f78a000cbb0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 shutdown_connections 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f789006b350 0x7f789006d810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78ac1603c0 0x7f78ac158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78ac0d1a50 0x7f78ac15e5d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 --2- 172.21.15.113:0/4195979034 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f78ac15bed0 0x7f78ac0cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.922+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 >> 172.21.15.113:0/4195979034 conn(0x7f78ac073130 msgr2=0x7f78ac0b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.923+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 shutdown_connections 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:47.923+0000 7f78b2bea700 1 -- 172.21.15.113:0/4195979034 wait complete. 2024-01-16T13:52:47.906 INFO:teuthology.orchestra.run.smithi113.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-16T13:52:47.918 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-16T13:52:47.946 INFO:tasks.ceph.mds.b.smithi124.stdout:starting mds.b at 2024-01-16T13:52:48.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.058+0000 7f95961cd700 1 Processor -- start 2024-01-16T13:52:48.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.058+0000 7f95961cd700 1 -- start start 2024-01-16T13:52:48.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f95961cd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f95900aec00 0x7f95900cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f95961cd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9590071b30 0x7f959006d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f95961cd700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900724b0 con 0x7f95900ce230 2024-01-16T13:52:48.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f95961cd700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95901568a0 con 0x7f95900aec00 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f95961cd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95900ce6e0 con 0x7f9590071b30 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f958effd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f95900ce230 0x7f9590071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53480/0 (socket says 172.21.15.113:53480) 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f958effd700 1 -- 172.21.15.113:0/1195858065 learned_addr learned my addr 172.21.15.113:0/1195858065 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.059+0000 7f958f7fe700 1 --2- 172.21.15.113:0/1195858065 >> v2:172.21.15.124:3300/0 conn(0x7f95900aec00 0x7f95900cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9590071b30 0x7f959006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480015029 0 0) 0x7f95900724b0 con 0x7f95900ce230 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95780035b0 con 0x7f95900ce230 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 419347836 0 0) 0x7f95780035b0 con 0x7f95900ce230 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9590071b30 msgr2=0x7f959006d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9590071b30 0x7f959006d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 >> v2:172.21.15.124:3300/0 conn(0x7f95900aec00 msgr2=0x7f95900cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 --2- 172.21.15.113:0/1195858065 >> v2:172.21.15.124:3300/0 conn(0x7f95900aec00 0x7f95900cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900d01b0 con 0x7f95900ce230 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f9580001670 con 0x7f95900ce230 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958f7fe700 1 --2- 172.21.15.113:0/1195858065 >> v2:172.21.15.124:3300/0 conn(0x7f95900aec00 0x7f95900cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 >> v1:172.21.15.113:6789/0 conn(0x7f95900ce230 legacy=0x7f9590071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9578003a00 0x7f9578003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9578004850 0x7f9578004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9578005210 0x7f957800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95901568a0 con 0x7f9578005210 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900ce6e0 con 0x7f9578003a00 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957800bf50 con 0x7f9578004850 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9590071b30 0x7f959006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958f7fe700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9578003a00 0x7f9578003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958effd700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9578004850 0x7f9578004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9578005210 0x7f957800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 -- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9578004850 msgr2=0x7f9578004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:48.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9578004850 0x7f9578004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 -- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9578003a00 msgr2=0x7f9578003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9578003a00 0x7f9578003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 -- 172.21.15.113:0/1195858065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780035b0 con 0x7f9578005210 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.060+0000 7f958ffff700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9578005210 0x7f957800ba10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f958800e250 tx=0x7f9578005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82686ff3a37c4832 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800ecf0 con 0x7f9578005210 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958800f040 con 0x7f9578005210 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f958e7fc700 1 -- 172.21.15.113:0/1195858065 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588003c40 con 0x7f9578005210 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f95961cd700 1 -- 172.21.15.113:0/1195858065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9578005210 msgr2=0x7f957800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f95961cd700 1 --2- 172.21.15.113:0/1195858065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9578005210 0x7f957800ba10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f958800e250 tx=0x7f9578005880 comp rx=0 tx=0).stop 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f958ffff700 1 -- 172.21.15.113:0/1195858065 reap_dead start 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f95961cd700 1 -- 172.21.15.113:0/1195858065 shutdown_connections 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f95961cd700 1 -- 172.21.15.113:0/1195858065 >> 172.21.15.113:0/1195858065 conn(0x7f9590074170 msgr2=0x7f959006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f95961cd700 1 -- 172.21.15.113:0/1195858065 shutdown_connections 2024-01-16T13:52:48.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.061+0000 7f95961cd700 1 -- 172.21.15.113:0/1195858065 wait complete. 2024-01-16T13:52:48.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.063+0000 7f95961cd700 1 Processor -- start 2024-01-16T13:52:48.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.063+0000 7f95961cd700 1 -- start start 2024-01-16T13:52:48.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f95961cd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9590149d70 0x7f95900ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f95961cd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 0x7f95900ab650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f95961cd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f95900abb90 0x7f95901585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f95961cd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900a98e0 con 0x7f959015b2f0 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f95961cd700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900ac010 con 0x7f9590149d70 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f95961cd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95901f1ee0 con 0x7f95900abb90 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 0x7f95900ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 0x7f95900ab650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56206/0 (socket says 172.21.15.113:56206) 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 -- 172.21.15.113:0/1510553084 learned_addr learned my addr 172.21.15.113:0/1510553084 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958f7fe700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9590149d70 0x7f95900ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958ffff700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f95900abb90 0x7f95901585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 -- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f95900abb90 msgr2=0x7f95901585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f95900abb90 0x7f95901585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 -- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9590149d70 msgr2=0x7f95900ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9590149d70 0x7f95900ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 -- 172.21.15.113:0/1510553084 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900add50 con 0x7f959015b2f0 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.064+0000 7f958effd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 0x7f95900ab650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9580007370 tx=0x7f9580007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.065+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000f660 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.065+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000f7e0 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.065+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95900ae030 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.065+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000fb90 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.065+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95900ae310 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.066+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95900a9a20 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.066+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f958000fcf0 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.067+0000 7f9577fff700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f957c06b350 0x7f957c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.067+0000 7f958f7fe700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f957c06b350 0x7f957c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.067+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9580089540 con 0x7f959015b2f0 2024-01-16T13:52:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.067+0000 7f958f7fe700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f957c06b350 0x7f957c06d810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9584009470 tx=0x7f9584009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:48.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.071+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95900a9a20 con 0x7f959015b2f0 2024-01-16T13:52:48.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.305+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f95900a9a20 con 0x7f959015b2f0 2024-01-16T13:52:48.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.924+0000 7f9577fff700 1 -- 172.21.15.113:0/1510553084 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f95900a9a20 con 0x7f959015b2f0 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f957c06b350 msgr2=0x7f957c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f957c06b350 0x7f957c06d810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9584009470 tx=0x7f9584009210 comp rx=0 tx=0).stop 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 msgr2=0x7f95900ab650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 0x7f95900ab650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9580007370 tx=0x7f9580007680 comp rx=0 tx=0).stop 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 shutdown_connections 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f957c06b350 0x7f957c06d810 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f95900abb90 0x7f95901585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9590149d70 0x7f95900ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 --2- 172.21.15.113:0/1510553084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f959015b2f0 0x7f95900ab650 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:48.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 >> 172.21.15.113:0/1510553084 conn(0x7f9590074170 msgr2=0x7f959015e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:48.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 shutdown_connections 2024-01-16T13:52:48.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:48.927+0000 7f95961cd700 1 -- 172.21.15.113:0/1510553084 wait complete. 2024-01-16T13:52:48.923 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-16T13:52:49.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.065+0000 7fc18b656700 1 Processor -- start 2024-01-16T13:52:49.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.065+0000 7fc18b656700 1 -- start start 2024-01-16T13:52:49.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc18b656700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc18415e0c0 0x7fc1840cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:49.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc18b656700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc184071d20 0x7fc1840b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc18b656700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1840d29e0 con 0x7fc1840d3a40 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc18b656700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840d2c80 con 0x7fc18415e0c0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc18b656700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840d2f60 con 0x7fc184071d20 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc18415e0c0 0x7fc1840cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc18415e0c0 0x7fc1840cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56216/0 (socket says 172.21.15.113:56216) 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 -- 172.21.15.113:0/8114774 learned_addr learned my addr 172.21.15.113:0/8114774 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc189bf3700 1 --2- 172.21.15.113:0/8114774 >> v2:172.21.15.124:3300/0 conn(0x7fc184071d20 0x7fc1840b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 -- 172.21.15.113:0/8114774 >> v2:172.21.15.124:3300/0 conn(0x7fc184071d20 msgr2=0x7fc1840b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 --2- 172.21.15.113:0/8114774 >> v2:172.21.15.124:3300/0 conn(0x7fc184071d20 0x7fc1840b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 -- 172.21.15.113:0/8114774 >> v1:172.21.15.124:6789/0 conn(0x7fc1840d3a40 legacy=0x7fc184071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 -- 172.21.15.113:0/8114774 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840d3240 con 0x7fc18415e0c0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.066+0000 7fc1893f2700 1 --2- 172.21.15.113:0/8114774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840cec80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc18408cf80 tx=0x7fc180009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d3fed62ab180ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc17bfff700 1 -- 172.21.15.113:0/8114774 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180011830 con 0x7fc18415e0c0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc17bfff700 1 -- 172.21.15.113:0/8114774 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180011990 con 0x7fc18415e0c0 2024-01-16T13:52:49.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc17bfff700 1 -- 172.21.15.113:0/8114774 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180011d20 con 0x7fc18415e0c0 2024-01-16T13:52:49.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 -- 172.21.15.113:0/8114774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 msgr2=0x7fc1840cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 --2- 172.21.15.113:0/8114774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840cec80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc18408cf80 tx=0x7fc180009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 -- 172.21.15.113:0/8114774 shutdown_connections 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 --2- 172.21.15.113:0/8114774 >> v2:172.21.15.124:3300/0 conn(0x7fc184071d20 0x7fc1840b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 --2- 172.21.15.113:0/8114774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840cec80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 -- 172.21.15.113:0/8114774 >> 172.21.15.113:0/8114774 conn(0x7fc18406b620 msgr2=0x7fc1840d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 -- 172.21.15.113:0/8114774 shutdown_connections 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.067+0000 7fc18b656700 1 -- 172.21.15.113:0/8114774 wait complete. 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.068+0000 7fc18b656700 1 Processor -- start 2024-01-16T13:52:49.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.068+0000 7fc18b656700 1 -- start start 2024-01-16T13:52:49.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc18b656700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc184071d20 0x7fc18415fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:49.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc18b656700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1840d3a40 0x7fc1840ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:49.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc18b656700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:49.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc18b656700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184147950 con 0x7fc18415e0c0 2024-01-16T13:52:49.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc18b656700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184147c30 con 0x7fc184071d20 2024-01-16T13:52:49.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc18b656700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc184147f10 con 0x7fc1840d3a40 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc189bf3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc189bf3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56226/0 (socket says 172.21.15.113:56226) 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc189bf3700 1 -- 172.21.15.113:0/1336512186 learned_addr learned my addr 172.21.15.113:0/1336512186 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc1893f2700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc184071d20 0x7fc18415fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.069+0000 7fc188bf1700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1840d3a40 0x7fc1840ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc189bf3700 1 -- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1840d3a40 msgr2=0x7fc1840ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc189bf3700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1840d3a40 0x7fc1840ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc189bf3700 1 -- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc184071d20 msgr2=0x7fc18415fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc189bf3700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc184071d20 0x7fc18415fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc189bf3700 1 -- 172.21.15.113:0/1336512186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc180009550 con 0x7fc18415e0c0 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc188bf1700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1840d3a40 0x7fc1840ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc1893f2700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc184071d20 0x7fc18415fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc189bf3700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840b7120 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00efc0 tx=0x7fc17c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00cd40 con 0x7fc18415e0c0 2024-01-16T13:52:49.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c00f040 con 0x7fc18415e0c0 2024-01-16T13:52:49.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c0146c0 con 0x7fc18415e0c0 2024-01-16T13:52:49.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18415e620 con 0x7fc18415e0c0 2024-01-16T13:52:49.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.070+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc184160410 con 0x7fc18415e0c0 2024-01-16T13:52:49.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.071+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc184067f60 con 0x7fc18415e0c0 2024-01-16T13:52:49.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.071+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc17c014820 con 0x7fc18415e0c0 2024-01-16T13:52:49.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.072+0000 7fc179ffb700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc16806b350 0x7fc16806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:49.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.072+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc17c08a530 con 0x7fc18415e0c0 2024-01-16T13:52:49.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.072+0000 7fc1893f2700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc16806b350 0x7fc16806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:49.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.072+0000 7fc1893f2700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc16806b350 0x7fc16806d810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc180014040 tx=0x7fc180008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:49.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.076+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc17c055340 con 0x7fc18415e0c0 2024-01-16T13:52:49.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.269+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc18415c350 con 0x7fc18415e0c0 2024-01-16T13:52:49.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.925+0000 7fc179ffb700 1 -- 172.21.15.113:0/1336512186 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc17c00cea0 con 0x7fc18415e0c0 2024-01-16T13:52:49.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc16806b350 msgr2=0x7fc16806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:49.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc16806b350 0x7fc16806d810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc180014040 tx=0x7fc180008030 comp rx=0 tx=0).stop 2024-01-16T13:52:49.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 msgr2=0x7fc1840b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:49.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840b7120 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00efc0 tx=0x7fc17c00c3a0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 shutdown_connections 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc16806b350 0x7fc16806d810 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc18415e0c0 0x7fc1840b7120 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc184071d20 0x7fc18415fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 --2- 172.21.15.113:0/1336512186 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc1840d3a40 0x7fc1840ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 >> 172.21.15.113:0/1336512186 conn(0x7fc18406b620 msgr2=0x7fc1840b80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 shutdown_connections 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:49.929+0000 7fc18b656700 1 -- 172.21.15.113:0/1336512186 wait complete. 2024-01-16T13:52:49.913 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_metadata' removed 2024-01-16T13:52:49.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-16T13:52:50.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.068+0000 7fc5024ed700 1 Processor -- start 2024-01-16T13:52:50.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.069+0000 7fc5024ed700 1 -- start start 2024-01-16T13:52:50.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.069+0000 7fc5024ed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc15df20 0x7fc4fc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:50.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.069+0000 7fc5024ed700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc06bed0 con 0x7fc4fc1622b0 2024-01-16T13:52:50.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.069+0000 7fc5024ed700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc06bba0 con 0x7fc4fc162820 2024-01-16T13:52:50.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.069+0000 7fc5024ed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc070060 con 0x7fc4fc15df20 2024-01-16T13:52:50.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc500a8a700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc4fc1622b0 0x7fc4fc0cc460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53492/0 (socket says 172.21.15.113:53492) 2024-01-16T13:52:50.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc500a8a700 1 -- 172.21.15.113:0/1270901401 learned_addr learned my addr 172.21.15.113:0/1270901401 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4fbfff700 1 --2- 172.21.15.113:0/1270901401 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc15df20 0x7fc4fc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603782087 0 0) 0x7fc4fc06bed0 con 0x7fc4fc1622b0 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e80035b0 con 0x7fc4fc1622b0 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356320348 0 0) 0x7fc4fc06bba0 con 0x7fc4fc162820 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4fc06bed0 con 0x7fc4fc162820 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4fbfff700 1 -- 172.21.15.113:0/1270901401 >> v1:172.21.15.124:6789/0 conn(0x7fc4fc162820 legacy=0x7fc4fc161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4fbfff700 1 -- 172.21.15.113:0/1270901401 >> v1:172.21.15.113:6789/0 conn(0x7fc4fc1622b0 legacy=0x7fc4fc0cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4fbfff700 1 -- 172.21.15.113:0/1270901401 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc06faa0 con 0x7fc4fc15df20 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4fbfff700 1 --2- 172.21.15.113:0/1270901401 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc15df20 0x7fc4fc06c360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009870 tx=0x7fc4ec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b58833752bd0ae6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec011830 con 0x7fc4fc15df20 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec011990 con 0x7fc4fc15df20 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.070+0000 7fc4faffd700 1 -- 172.21.15.113:0/1270901401 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec011d00 con 0x7fc4fc15df20 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 -- 172.21.15.113:0/1270901401 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc15df20 msgr2=0x7fc4fc06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 --2- 172.21.15.113:0/1270901401 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc15df20 0x7fc4fc06c360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009870 tx=0x7fc4ec009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:50.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 -- 172.21.15.113:0/1270901401 shutdown_connections 2024-01-16T13:52:50.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 --2- 172.21.15.113:0/1270901401 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc15df20 0x7fc4fc06c360 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 -- 172.21.15.113:0/1270901401 >> 172.21.15.113:0/1270901401 conn(0x7fc4fc06dc80 msgr2=0x7fc4fc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:50.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 -- 172.21.15.113:0/1270901401 shutdown_connections 2024-01-16T13:52:50.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.071+0000 7fc5024ed700 1 -- 172.21.15.113:0/1270901401 wait complete. 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.072+0000 7fc5024ed700 1 Processor -- start 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.072+0000 7fc5024ed700 1 -- start start 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc5024ed700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 0x7fc4fc1e7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc5024ed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc1622b0 0x7fc4fc0d0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc5024ed700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4fc162820 0x7fc4fc146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc5024ed700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0d0c40 con 0x7fc4fc15df20 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc5024ed700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0d0ea0 con 0x7fc4fc162820 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc5024ed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0d1180 con 0x7fc4fc1622b0 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 0x7fc4fc1e7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 0x7fc4fc1e7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56228/0 (socket says 172.21.15.113:56228) 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 -- 172.21.15.113:0/2248613363 learned_addr learned my addr 172.21.15.113:0/2248613363 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fb7fe700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc1622b0 0x7fc4fc0d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc500a8a700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4fc162820 0x7fc4fc146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:50.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 -- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc1622b0 msgr2=0x7fc4fc0d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc1622b0 0x7fc4fc0d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 -- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4fc162820 msgr2=0x7fc4fc146380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4fc162820 0x7fc4fc146380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fbfff700 1 -- 172.21.15.113:0/2248613363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0094a0 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc500a8a700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4fc162820 0x7fc4fc146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.073+0000 7fc4fb7fe700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc1622b0 0x7fc4fc0d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.074+0000 7fc4fbfff700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 0x7fc4fc1e7cb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0090b0 tx=0x7fc4ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.074+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0045c0 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.074+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec01f070 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.074+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc0d1460 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.074+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec003900 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.074+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc1468c0 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.075+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4fc06f730 con 0x7fc4fc15df20 2024-01-16T13:52:50.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.075+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc4ec003a60 con 0x7fc4fc15df20 2024-01-16T13:52:50.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.075+0000 7fc4f8ff9700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc4dc06b350 0x7fc4dc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:50.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.076+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fc4ec021070 con 0x7fc4fc15df20 2024-01-16T13:52:50.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.076+0000 7fc4fb7fe700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc4dc06b350 0x7fc4dc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:50.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.076+0000 7fc4fb7fe700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc4dc06b350 0x7fc4dc06d810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0b7ce0 tx=0x7fc4e4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:50.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.080+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4ec056200 con 0x7fc4fc15df20 2024-01-16T13:52:50.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.270+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc4fc067f60 con 0x7fc4fc15df20 2024-01-16T13:52:50.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.931+0000 7fc4f8ff9700 1 -- 172.21.15.113:0/2248613363 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc4ec006cc0 con 0x7fc4fc15df20 2024-01-16T13:52:50.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc4dc06b350 msgr2=0x7fc4dc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc4dc06b350 0x7fc4dc06d810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0b7ce0 tx=0x7fc4e4009040 comp rx=0 tx=0).stop 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 msgr2=0x7fc4fc1e7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 0x7fc4fc1e7cb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0090b0 tx=0x7fc4ec004750 comp rx=0 tx=0).stop 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 shutdown_connections 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc4dc06b350 0x7fc4dc06d810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4fc162820 0x7fc4fc146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4fc15df20 0x7fc4fc1e7cb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 --2- 172.21.15.113:0/2248613363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4fc1622b0 0x7fc4fc0d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 >> 172.21.15.113:0/2248613363 conn(0x7fc4fc06dc80 msgr2=0x7fc4fc0b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 shutdown_connections 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:50.934+0000 7fc5024ed700 1 -- 172.21.15.113:0/2248613363 wait complete. 2024-01-16T13:52:50.918 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_data' removed 2024-01-16T13:52:50.932 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T13:52:50.932 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T13:52:50.932 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T13:52:50.933 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T13:52:50.933 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T13:52:50.933 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T13:52:50.933 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T13:52:50.933 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T13:52:50.933 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T13:52:50.933 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T13:52:50.933 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T13:52:50.934 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T13:52:50.934 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T13:52:50.934 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T13:52:50.934 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T13:52:50.934 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T13:52:50.943 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T13:52:50.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T13:52:50.944 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T13:52:50.944 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T13:52:50.946 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T13:52:50.946 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T13:52:50.947 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T13:52:50.947 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T13:52:50.985 INFO:tasks.ceph.mds.a:Started 2024-01-16T13:52:50.985 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:50.986 INFO:tasks.ceph.mds.c:Started 2024-01-16T13:52:50.986 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:50.989 INFO:tasks.ceph.mds.b:Started 2024-01-16T13:52:50.989 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:50.990 INFO:tasks.ceph.mds.d:Started 2024-01-16T13:52:50.990 DEBUG:teuthology.parallel:result is None 2024-01-16T13:52:50.990 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T13:52:51.068 INFO:tasks.ceph.mds.c.smithi113.stdout:starting mds.c at 2024-01-16T13:52:51.068 INFO:tasks.ceph.mds.a.smithi113.stdout:starting mds.a at 2024-01-16T13:52:51.077 INFO:tasks.ceph.mds.b.smithi124.stdout:starting mds.b at 2024-01-16T13:52:51.078 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2024-01-16T13:52:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.137+0000 7fc299a53700 1 Processor -- start 2024-01-16T13:52:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.137+0000 7fc299a53700 1 -- start start 2024-01-16T13:52:51.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc299a53700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2940d0870 0x7fc2940d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc299a53700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2940d1210 con 0x7fc2940b6ef0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc299a53700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2940d20b0 con 0x7fc29415e1e0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc299a53700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940d3150 con 0x7fc2940d0870 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc2937fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc2940b6ef0 0x7fc2940b7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53510/0 (socket says 172.21.15.113:53510) 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc2937fe700 1 -- 172.21.15.113:0/4097215829 learned_addr learned my addr 172.21.15.113:0/4097215829 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.138+0000 7fc2927fc700 1 --2- 172.21.15.113:0/4097215829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2940d0870 0x7fc2940d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc291ffb700 1 -- 172.21.15.113:0/4097215829 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298915290 0 0) 0x7fc2940d1210 con 0x7fc2940b6ef0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc291ffb700 1 -- 172.21.15.113:0/4097215829 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc27c0035b0 con 0x7fc2940b6ef0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc291ffb700 1 -- 172.21.15.113:0/4097215829 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213673511 0 0) 0x7fc2940d20b0 con 0x7fc29415e1e0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc2927fc700 1 -- 172.21.15.113:0/4097215829 >> v1:172.21.15.124:6789/0 conn(0x7fc29415e1e0 legacy=0x7fc2940d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc2927fc700 1 -- 172.21.15.113:0/4097215829 >> v1:172.21.15.113:6789/0 conn(0x7fc2940b6ef0 legacy=0x7fc2940b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc2927fc700 1 -- 172.21.15.113:0/4097215829 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940d2390 con 0x7fc2940d0870 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc2927fc700 1 --2- 172.21.15.113:0/4097215829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2940d0870 0x7fc2940d0cd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc284009870 tx=0x7fc284009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64a61a6417c3c66 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc291ffb700 1 -- 172.21.15.113:0/4097215829 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284011830 con 0x7fc2940d0870 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc291ffb700 1 -- 172.21.15.113:0/4097215829 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284011990 con 0x7fc2940d0870 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.139+0000 7fc291ffb700 1 -- 172.21.15.113:0/4097215829 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284011d20 con 0x7fc2940d0870 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 -- 172.21.15.113:0/4097215829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2940d0870 msgr2=0x7fc2940d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 --2- 172.21.15.113:0/4097215829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2940d0870 0x7fc2940d0cd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc284009870 tx=0x7fc284009b80 comp rx=0 tx=0).stop 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 -- 172.21.15.113:0/4097215829 shutdown_connections 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 --2- 172.21.15.113:0/4097215829 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc2940d0870 0x7fc2940d0cd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 -- 172.21.15.113:0/4097215829 >> 172.21.15.113:0/4097215829 conn(0x7fc2940b5b50 msgr2=0x7fc2940b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 -- 172.21.15.113:0/4097215829 shutdown_connections 2024-01-16T13:52:51.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.140+0000 7fc299a53700 1 -- 172.21.15.113:0/4097215829 wait complete. 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.141+0000 7fc299a53700 1 Processor -- start 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.141+0000 7fc299a53700 1 -- start start 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc2940b6ef0 0x7fc2940cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 0x7fc294152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc29415e1e0 0x7fc294157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc294152940 con 0x7fc2940d0870 2024-01-16T13:52:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940cdeb0 con 0x7fc2940b6ef0 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc294159920 con 0x7fc29415e1e0 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 0x7fc294152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 0x7fc294152400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56266/0 (socket says 172.21.15.113:56266) 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc292ffd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc2940b6ef0 0x7fc2940cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 -- 172.21.15.113:0/1942471720 learned_addr learned my addr 172.21.15.113:0/1942471720 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2937fe700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc29415e1e0 0x7fc294157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 -- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc29415e1e0 msgr2=0x7fc294157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc29415e1e0 0x7fc294157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 -- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc2940b6ef0 msgr2=0x7fc2940cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc2940b6ef0 0x7fc2940cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 -- 172.21.15.113:0/1942471720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840094a0 con 0x7fc2940d0870 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2937fe700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc29415e1e0 0x7fc294157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc292ffd700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc2940b6ef0 0x7fc2940cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc2927fc700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 0x7fc294152400 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc284009bc0 tx=0x7fc284004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28401a7e0 con 0x7fc2940d0870 2024-01-16T13:52:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28401a940 con 0x7fc2940d0870 2024-01-16T13:52:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc294159c00 con 0x7fc2940d0870 2024-01-16T13:52:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.142+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28401acf0 con 0x7fc2940d0870 2024-01-16T13:52:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.143+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29415a1c0 con 0x7fc2940d0870 2024-01-16T13:52:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.144+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29415f800 con 0x7fc2940d0870 2024-01-16T13:52:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.144+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc284022070 con 0x7fc2940d0870 2024-01-16T13:52:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.145+0000 7fc27bfff700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc28006b350 0x7fc28006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.145+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc28408a990 con 0x7fc2940d0870 2024-01-16T13:52:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.145+0000 7fc292ffd700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc28006b350 0x7fc28006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.145+0000 7fc292ffd700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc28006b350 0x7fc28006d810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc288000f60 tx=0x7fc288009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:51.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.149+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc29415f800 con 0x7fc2940d0870 2024-01-16T13:52:51.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-01-16T13:52:51.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-16T13:52:51.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-16T13:52:51.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:51.354+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc29415f800 con 0x7fc2940d0870 2024-01-16T13:52:52.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.551+0000 7fc27bfff700 1 -- 172.21.15.113:0/1942471720 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc29415f800 con 0x7fc2940d0870 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc28006b350 msgr2=0x7fc28006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc28006b350 0x7fc28006d810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc288000f60 tx=0x7fc288009040 comp rx=0 tx=0).stop 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 msgr2=0x7fc294152400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 0x7fc294152400 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc284009bc0 tx=0x7fc284004750 comp rx=0 tx=0).stop 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 shutdown_connections 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc28006b350 0x7fc28006d810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc29415e1e0 0x7fc294157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc2940b6ef0 0x7fc2940cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 --2- 172.21.15.113:0/1942471720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2940d0870 0x7fc294152400 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 >> 172.21.15.113:0/1942471720 conn(0x7fc2940b5b50 msgr2=0x7fc2940b75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:52.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 shutdown_connections 2024-01-16T13:52:52.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.554+0000 7fc299a53700 1 -- 172.21.15.113:0/1942471720 wait complete. 2024-01-16T13:52:52.538 INFO:teuthology.orchestra.run.smithi113.stderr: removed all blocklist entries 2024-01-16T13:52:52.551 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T13:52:52.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.692+0000 7f3d58795700 1 Processor -- start 2024-01-16T13:52:52.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.692+0000 7f3d58795700 1 -- start start 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d58795700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d5015e2a0 0x7f3d5006ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d58795700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3d500ce090 0x7f3d5006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d58795700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5006e7f0 con 0x7f3d50073f90 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d58795700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5006dec0 con 0x7f3d500ce090 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d58795700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5006e1a0 con 0x7f3d5015e2a0 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d55d30700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3d500ce090 0x7f3d5006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d55d30700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3d500ce090 0x7f3d5006ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56272/0 (socket says 172.21.15.113:56272) 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.693+0000 7f3d55d30700 1 -- 172.21.15.113:0/3206003032 learned_addr learned my addr 172.21.15.113:0/3206003032 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d56531700 1 --2- 172.21.15.113:0/3206003032 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d5015e2a0 0x7f3d5006ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d55d30700 1 -- 172.21.15.113:0/3206003032 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d5015e2a0 msgr2=0x7f3d5006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:52.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d55d30700 1 --2- 172.21.15.113:0/3206003032 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d5015e2a0 0x7f3d5006ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d55d30700 1 -- 172.21.15.113:0/3206003032 >> v1:172.21.15.124:6789/0 conn(0x7f3d50073f90 legacy=0x7f3d500cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d55d30700 1 -- 172.21.15.113:0/3206003032 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5006e480 con 0x7f3d500ce090 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d55d30700 1 --2- 172.21.15.113:0/3206003032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d500ce090 0x7f3d5006ec80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d400098f0 tx=0x7f3d40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d755be3f1f4da4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d5552f700 1 -- 172.21.15.113:0/3206003032 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40004230 con 0x7f3d500ce090 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d5552f700 1 -- 172.21.15.113:0/3206003032 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40004390 con 0x7f3d500ce090 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.694+0000 7f3d5552f700 1 -- 172.21.15.113:0/3206003032 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d400036a0 con 0x7f3d500ce090 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 -- 172.21.15.113:0/3206003032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d500ce090 msgr2=0x7f3d5006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 --2- 172.21.15.113:0/3206003032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d500ce090 0x7f3d5006ec80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d400098f0 tx=0x7f3d40009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 -- 172.21.15.113:0/3206003032 shutdown_connections 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 --2- 172.21.15.113:0/3206003032 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d5015e2a0 0x7f3d5006ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 --2- 172.21.15.113:0/3206003032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d500ce090 0x7f3d5006ec80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 -- 172.21.15.113:0/3206003032 >> 172.21.15.113:0/3206003032 conn(0x7f3d500cd570 msgr2=0x7f3d500cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 -- 172.21.15.113:0/3206003032 shutdown_connections 2024-01-16T13:52:52.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.695+0000 7f3d58795700 1 -- 172.21.15.113:0/3206003032 wait complete. 2024-01-16T13:52:52.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.696+0000 7f3d58795700 1 Processor -- start 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.696+0000 7f3d58795700 1 -- start start 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 0x7f3d5014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d500ce090 0x7f3d501f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d5015e2a0 0x7f3d5015a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d50159c80 con 0x7f3d50073f90 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5015ea50 con 0x7f3d5015e2a0 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5015ed30 con 0x7f3d500ce090 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 0x7f3d5014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 0x7f3d5014b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56278/0 (socket says 172.21.15.113:56278) 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 -- 172.21.15.113:0/1997572404 learned_addr learned my addr 172.21.15.113:0/1997572404 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d55d30700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d500ce090 0x7f3d501f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56d32700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d5015e2a0 0x7f3d5015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 -- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d500ce090 msgr2=0x7f3d501f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d500ce090 0x7f3d501f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 -- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d5015e2a0 msgr2=0x7f3d5015a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d5015e2a0 0x7f3d5015a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 -- 172.21.15.113:0/1997572404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40009550 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56d32700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d5015e2a0 0x7f3d5015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d56531700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 0x7f3d5014b650 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c7f0 tx=0x7f3d4c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c004220 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c004380 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5015f010 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c0036a0 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.697+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d501f51f0 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.698+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d50067f60 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.699+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3d4c003800 con 0x7f3d50073f90 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.699+0000 7f3d46ffd700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3d3406b350 0x7f3d3406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:52.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.699+0000 7f3d55d30700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3d3406b350 0x7f3d3406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:52.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.699+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d4c089410 con 0x7f3d50073f90 2024-01-16T13:52:52.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.700+0000 7f3d55d30700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3d3406b350 0x7f3d3406d810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009130 tx=0x7f3d40008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:52.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.704+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d4c0545e0 con 0x7f3d50073f90 2024-01-16T13:52:52.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.978+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3d50072e00 con 0x7f3d50073f90 2024-01-16T13:52:52.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.979+0000 7f3d46ffd700 1 -- 172.21.15.113:0/1997572404 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f3d4c057c00 con 0x7f3d50073f90 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "auth_dump": [ 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.a", 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlyDrJARAAz//i2ClH4sbEANRxAiIMlQ==", 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.962 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.b", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlMdI0DhAAzYRcAH7Jyx48yzJ3Lz3K/g==", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.c", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlwIsPCRAA4q2vkMfBy92/h9eg1gfqRA==", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.963 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mds.d", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlioyRFRAAlUs/AAsXtP0itYJGyIgSYg==", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.964 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.0", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDUiaZl8s3FNBAA09pce2MNSApF+drjMAkNPA==", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.1", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDXiaZlxn/oMxAA5chIuJyqCbNKYh9A1Ta3PQ==", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.965 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.2", 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDaiaZl5hgvMhAAHSCDsDeu8fseIF0mJu4Ctg==", 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.3", 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDdiaZlDZJfMBAAER2AYSt+Qk/a0dwa2/Ij2w==", 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.966 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.4", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDjiaZlHMALMxAAzgDDZvHvDA+wtSH4Lfrh7Q==", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.5", 2024-01-16T13:52:52.967 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDmiaZl4Fk6MhAAlK70uyn3b6DFsjXbBsM0Mw==", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.6", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDpiaZlur1mMRAAzpfPmm4wDxD4Kcex/eBOsA==", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.968 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "osd.7", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDsiaZlMtGZMBAAI9uyUo5CajqfhcKH2/xxnQ==", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow profile osd", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile osd", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.0", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlwFPiGRAAc/ztvEiVlF5DA4U3NZB3Og==", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.969 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.1", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlxvH/IhAANuKb2btH+PF7v4wMKZaSxw==", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.2", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlNXyXHxAAx9VCUKHDRhkvHMC+VRxGSQ==", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.3", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDRiaZlT+ZVJhAABoClp/1hNEv4UJ5Drk3JlQ==", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow", 2024-01-16T13:52:52.971 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow r", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow rw", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow rwx" 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.admin", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDQiaZl/k7uGRAAsns9a4Y6uLGOjYDqLIZQew==", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow *", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": "allow *", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow *", 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.972 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-mds", 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDyiaZlVuktLRAAtIOnpIZ+d/ks9lXzKLqW5w==", 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-mds" 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-mgr", 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDyiaZl6PMtLRAAr8jImGASYqFsiZU/7S8gGg==", 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-osd", 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDyiaZlp/stLRAANRPD5wt3Vi9YkSXnd2XF9w==", 2024-01-16T13:52:52.973 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-osd" 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-rbd", 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDyiaZlOQMuLRAAGC7C5rySzFOx/u+lpyUloQ==", 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDyiaZlpgouLRAAqV9ZNukJy/bIYp7NRBTvRg==", 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-16T13:52:52.974 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "client.bootstrap-rgw", 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDyiaZlghIuLRAA4Y0DYAEJtg9t1WiaQR4bYw==", 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mgr.x", 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDQiaZlsUoIOhAAlHDQ25EXkUJbqP1AAdaJzw==", 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow *", 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile mgr", 2024-01-16T13:52:52.975 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: "entity": "mgr.y", 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: "key": "AQDQiaZlJxhjNhAA83zCf3EwB5xwr7O948uvLQ==", 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": { 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": "allow *", 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": "allow profile mgr", 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": "allow *" 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-01-16T13:52:52.976 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3d3406b350 msgr2=0x7f3d3406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3d3406b350 0x7f3d3406d810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009130 tx=0x7f3d40008030 comp rx=0 tx=0).stop 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 msgr2=0x7f3d5014b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 0x7f3d5014b650 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c7f0 tx=0x7f3d4c00cb00 comp rx=0 tx=0).stop 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 shutdown_connections 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3d3406b350 0x7f3d3406d810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d5015e2a0 0x7f3d5015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d50073f90 0x7f3d5014b650 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 --2- 172.21.15.113:0/1997572404 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d500ce090 0x7f3d501f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 >> 172.21.15.113:0/1997572404 conn(0x7f3d500cd570 msgr2=0x7f3d500b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 shutdown_connections 2024-01-16T13:52:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:52.981+0000 7f3d58795700 1 -- 172.21.15.113:0/1997572404 wait complete. 2024-01-16T13:52:52.979 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-16T13:52:53.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 Processor -- start 2024-01-16T13:52:53.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 -- start start 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8ec15ddf0 0x7fd8ec0b1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd8ec072df0 0x7fd8ec0cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0ccde0 con 0x7fd8ec0b0230 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0cd0c0 con 0x7fd8ec15ddf0 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8f11e8700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0cd3a0 con 0x7fd8ec072df0 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8ec15ddf0 0x7fd8ec0b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8ec15ddf0 0x7fd8ec0b1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56282/0 (socket says 172.21.15.113:56282) 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 -- 172.21.15.113:0/1120677433 learned_addr learned my addr 172.21.15.113:0/1120677433 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8eb59e700 1 --2- 172.21.15.113:0/1120677433 >> v2:172.21.15.124:3300/0 conn(0x7fd8ec072df0 0x7fd8ec0cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 -- 172.21.15.113:0/1120677433 >> v2:172.21.15.124:3300/0 conn(0x7fd8ec072df0 msgr2=0x7fd8ec0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 --2- 172.21.15.113:0/1120677433 >> v2:172.21.15.124:3300/0 conn(0x7fd8ec072df0 0x7fd8ec0cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 -- 172.21.15.113:0/1120677433 >> v1:172.21.15.124:6789/0 conn(0x7fd8ec0b0230 legacy=0x7fd8ec0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 -- 172.21.15.113:0/1120677433 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0cd680 con 0x7fd8ec15ddf0 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8ead9d700 1 --2- 172.21.15.113:0/1120677433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec0b1320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0cddf0 tx=0x7fd8dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138de03cc0c25e58 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8e9d9b700 1 -- 172.21.15.113:0/1120677433 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc00b870 con 0x7fd8ec15ddf0 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8e9d9b700 1 -- 172.21.15.113:0/1120677433 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc00b9d0 con 0x7fd8ec15ddf0 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.120+0000 7fd8e9d9b700 1 -- 172.21.15.113:0/1120677433 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc00bd60 con 0x7fd8ec15ddf0 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 -- 172.21.15.113:0/1120677433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec15ddf0 msgr2=0x7fd8ec0b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/1120677433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec0b1320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0cddf0 tx=0x7fd8dc009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 -- 172.21.15.113:0/1120677433 shutdown_connections 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/1120677433 >> v2:172.21.15.124:3300/0 conn(0x7fd8ec072df0 0x7fd8ec0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/1120677433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec0b1320 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 -- 172.21.15.113:0/1120677433 >> 172.21.15.113:0/1120677433 conn(0x7fd8ec0b57b0 msgr2=0x7fd8ec0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 -- 172.21.15.113:0/1120677433 shutdown_connections 2024-01-16T13:52:53.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.121+0000 7fd8f11e8700 1 -- 172.21.15.113:0/1120677433 wait complete. 2024-01-16T13:52:53.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.122+0000 7fd8f11e8700 1 Processor -- start 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.122+0000 7fd8f11e8700 1 -- start start 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8f11e8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8ec072df0 0x7fd8ec09a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8f11e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 0x7fd8ec0ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8f11e8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec1677b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8f11e8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec14d0e0 con 0x7fd8ec0b0230 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8f11e8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec09d450 con 0x7fd8ec15ddf0 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8f11e8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec153b60 con 0x7fd8ec072df0 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 0x7fd8ec0ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 0x7fd8ec0ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56284/0 (socket says 172.21.15.113:56284) 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 -- 172.21.15.113:0/606910110 learned_addr learned my addr 172.21.15.113:0/606910110 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8eb59e700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec1677b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ead9d700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8ec072df0 0x7fd8ec09a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 -- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8ec072df0 msgr2=0x7fd8ec09a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8ec072df0 0x7fd8ec09a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 -- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8ec15ddf0 msgr2=0x7fd8ec1677b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec1677b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.123+0000 7fd8ea59c700 1 -- 172.21.15.113:0/606910110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc009550 con 0x7fd8ec0b0230 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.124+0000 7fd8ea59c700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 0x7fd8ec0ae660 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c4b0 tx=0x7fd8e000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.124+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00052d0 con 0x7fd8ec0b0230 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.124+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0005430 con 0x7fd8ec0b0230 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.124+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec09ce00 con 0x7fd8ec0b0230 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.124+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00038c0 con 0x7fd8ec0b0230 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.124+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec153e40 con 0x7fd8ec0b0230 2024-01-16T13:52:53.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.125+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec09cc60 con 0x7fd8ec0b0230 2024-01-16T13:52:53.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.125+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd8e0003a20 con 0x7fd8ec0b0230 2024-01-16T13:52:53.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.125+0000 7fd8d37fe700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8d806b350 0x7fd8d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.126+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd8e0088d00 con 0x7fd8ec0b0230 2024-01-16T13:52:53.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.126+0000 7fd8ead9d700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8d806b350 0x7fd8d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.126+0000 7fd8ead9d700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8d806b350 0x7fd8d806d810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009c10 tx=0x7fd8dc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.131+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8ec09cc60 con 0x7fd8ec0b0230 2024-01-16T13:52:53.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.400+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fd8ec09cc60 con 0x7fd8ec0b0230 2024-01-16T13:52:53.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.403+0000 7fd8d37fe700 1 -- 172.21.15.113:0/606910110 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd8ec09cc60 con 0x7fd8ec0b0230 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8d806b350 msgr2=0x7fd8d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8d806b350 0x7fd8d806d810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009c10 tx=0x7fd8dc008030 comp rx=0 tx=0).stop 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 msgr2=0x7fd8ec0ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 0x7fd8ec0ae660 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c4b0 tx=0x7fd8e000c870 comp rx=0 tx=0).stop 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 shutdown_connections 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8d806b350 0x7fd8d806d810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8ec15ddf0 0x7fd8ec1677b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8ec072df0 0x7fd8ec09a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 --2- 172.21.15.113:0/606910110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8ec0b0230 0x7fd8ec0ae660 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 >> 172.21.15.113:0/606910110 conn(0x7fd8ec0b57b0 msgr2=0x7fd8ec0cf090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:53.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 shutdown_connections 2024-01-16T13:52:53.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.405+0000 7fd8f11e8700 1 -- 172.21.15.113:0/606910110 wait complete. 2024-01-16T13:52:53.400 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-16T13:52:53.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.539+0000 7f28239ce700 1 Processor -- start 2024-01-16T13:52:53.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.539+0000 7f28239ce700 1 -- start start 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f28239ce700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f28239ce700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 0x7f281c074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f28239ce700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c06e7f0 con 0x7f281c0ce090 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f28239ce700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c074960 con 0x7f281c073f90 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f28239ce700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c06dec0 con 0x7f281c15e2a0 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f2821f6b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 0x7f281c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f282176a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f2820f69700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f281c0ce090 0x7f281c06ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:35596/0 (socket says 172.21.15.113:35596) 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f2821f6b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 0x7f281c074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:39674/0 (socket says 172.21.15.113:39674) 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f2820f69700 1 -- 172.21.15.113:0/4267077866 learned_addr learned my addr 172.21.15.113:0/4267077866 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f282176a700 1 -- 172.21.15.113:0/4267077866 >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 msgr2=0x7f281c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f282176a700 1 --2- 172.21.15.113:0/4267077866 >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 0x7f281c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f282176a700 1 -- 172.21.15.113:0/4267077866 >> v1:172.21.15.124:6789/0 conn(0x7f281c0ce090 legacy=0x7f281c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f282176a700 1 -- 172.21.15.113:0/4267077866 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c06e1a0 con 0x7f281c15e2a0 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f2821f6b700 1 --2- 172.21.15.113:0/4267077866 >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 0x7f281c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.540+0000 7f282176a700 1 --2- 172.21.15.113:0/4267077866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c06ba70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f280c009840 tx=0x7f280c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d078176fba1abe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f2813fff700 1 -- 172.21.15.113:0/4267077866 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c011830 con 0x7f281c15e2a0 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f2813fff700 1 -- 172.21.15.113:0/4267077866 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c011990 con 0x7f281c15e2a0 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f2813fff700 1 -- 172.21.15.113:0/4267077866 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c011d20 con 0x7f281c15e2a0 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 -- 172.21.15.113:0/4267077866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 msgr2=0x7f281c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 --2- 172.21.15.113:0/4267077866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c06ba70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f280c009840 tx=0x7f280c009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 -- 172.21.15.113:0/4267077866 shutdown_connections 2024-01-16T13:52:53.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 --2- 172.21.15.113:0/4267077866 >> v2:172.21.15.124:3300/0 conn(0x7f281c073f90 0x7f281c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 --2- 172.21.15.113:0/4267077866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c06ba70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 -- 172.21.15.113:0/4267077866 >> 172.21.15.113:0/4267077866 conn(0x7f281c0cd570 msgr2=0x7f281c0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:53.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 -- 172.21.15.113:0/4267077866 shutdown_connections 2024-01-16T13:52:53.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.541+0000 7f28239ce700 1 -- 172.21.15.113:0/4267077866 wait complete. 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.542+0000 7f28239ce700 1 Processor -- start 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 -- start start 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f281c073f90 0x7f281c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 0x7f281c0ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c1550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0addb0 con 0x7f281c0ce090 2024-01-16T13:52:53.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0adfe0 con 0x7f281c073f90 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f28239ce700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0ae2c0 con 0x7f281c15e2a0 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f2820f69700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 0x7f281c0ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f2820f69700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 0x7f281c0ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48758/0 (socket says 172.21.15.113:48758) 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.543+0000 7f2820f69700 1 -- 172.21.15.113:0/1081062292 learned_addr learned my addr 172.21.15.113:0/1081062292 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2821f6b700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c1550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f282176a700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f281c073f90 0x7f281c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2820f69700 1 -- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 msgr2=0x7f281c1550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2820f69700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c1550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2820f69700 1 -- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f281c073f90 msgr2=0x7f281c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2820f69700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f281c073f90 0x7f281c158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2820f69700 1 -- 172.21.15.113:0/1081062292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2814009a70 con 0x7f281c0ce090 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2820f69700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 0x7f281c0ad870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f281800c520 tx=0x7f281800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818005d70 con 0x7f281c0ce090 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28180070d0 con 0x7f281c0ce090 2024-01-16T13:52:53.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c0094a0 con 0x7f281c0ce090 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818007440 con 0x7f281c0ce090 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.544+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c0ae880 con 0x7f281c0ce090 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.545+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c0689e0 con 0x7f281c0ce090 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.546+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f28180075a0 con 0x7f281c0ce090 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.546+0000 7f2811ffb700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f280006b350 0x7f280006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.546+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2818088ab0 con 0x7f281c0ce090 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.546+0000 7f282176a700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f280006b350 0x7f280006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.546+0000 7f282176a700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f280006b350 0x7f280006d810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f280c000c00 tx=0x7f280c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.550+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2818053d30 con 0x7f281c0ce090 2024-01-16T13:52:53.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.821+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f281c067f60 con 0x7f281c0ce090 2024-01-16T13:52:53.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.823+0000 7f2811ffb700 1 -- 172.21.15.113:0/1081062292 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f281800ce60 con 0x7f281c0ce090 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f280006b350 msgr2=0x7f280006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f280006b350 0x7f280006d810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f280c000c00 tx=0x7f280c008030 comp rx=0 tx=0).stop 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 msgr2=0x7f281c0ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 0x7f281c0ad870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f281800c520 tx=0x7f281800c830 comp rx=0 tx=0).stop 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 shutdown_connections 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f280006b350 0x7f280006d810 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f281c15e2a0 0x7f281c1550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f281c073f90 0x7f281c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 --2- 172.21.15.113:0/1081062292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f281c0ce090 0x7f281c0ad870 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 >> 172.21.15.113:0/1081062292 conn(0x7f281c0cd570 msgr2=0x7f281c0b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 shutdown_connections 2024-01-16T13:52:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.826+0000 7f28239ce700 1 -- 172.21.15.113:0/1081062292 wait complete. 2024-01-16T13:52:53.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-16T13:52:53.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.961+0000 7f0e88d3e700 1 Processor -- start 2024-01-16T13:52:53.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.961+0000 7f0e88d3e700 1 -- start start 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0e8415af10 0x7f0e8415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415f630 0x7f0e8406b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e841414c0 con 0x7f0e840ce140 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8415c300 con 0x7f0e8415af10 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840ce9f0 con 0x7f0e8415f630 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0e8415af10 0x7f0e8415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0e8415af10 0x7f0e8415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48768/0 (socket says 172.21.15.113:48768) 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 -- 172.21.15.113:0/785631899 learned_addr learned my addr 172.21.15.113:0/785631899 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e82d9d700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415f630 0x7f0e8406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 -- 172.21.15.113:0/785631899 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415f630 msgr2=0x7f0e8406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415f630 0x7f0e8406b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 -- 172.21.15.113:0/785631899 >> v1:172.21.15.124:6789/0 conn(0x7f0e840ce140 legacy=0x7f0e8415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 -- 172.21.15.113:0/785631899 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e840cecd0 con 0x7f0e8415af10 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e82d9d700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415f630 0x7f0e8406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8259c700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e8415e9e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400bb90 tx=0x7f0e7400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5834c0407da1f538 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8159a700 1 -- 172.21.15.113:0/785631899 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400dd10 con 0x7f0e8415af10 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8159a700 1 -- 172.21.15.113:0/785631899 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7400de70 con 0x7f0e8415af10 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e8159a700 1 -- 172.21.15.113:0/785631899 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74004230 con 0x7f0e8415af10 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- 172.21.15.113:0/785631899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 msgr2=0x7f0e8415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e8415e9e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400bb90 tx=0x7f0e7400bea0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- 172.21.15.113:0/785631899 shutdown_connections 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415f630 0x7f0e8406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/785631899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e8415e9e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- 172.21.15.113:0/785631899 >> 172.21.15.113:0/785631899 conn(0x7f0e840cd570 msgr2=0x7f0e840cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- 172.21.15.113:0/785631899 shutdown_connections 2024-01-16T13:52:53.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.962+0000 7f0e88d3e700 1 -- 172.21.15.113:0/785631899 wait complete. 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.964+0000 7f0e88d3e700 1 Processor -- start 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.964+0000 7f0e88d3e700 1 -- start start 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e840ce140 0x7f0e8413d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e84158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415b2f0 0x7f0e8415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8413dbb0 con 0x7f0e8415af10 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8415bce0 con 0x7f0e840ce140 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e84158e70 con 0x7f0e8415b2f0 2024-01-16T13:52:53.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e84158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e84158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48780/0 (socket says 172.21.15.113:48780) 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 -- 172.21.15.113:0/3038658195 learned_addr learned my addr 172.21.15.113:0/3038658195 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e8259c700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e840ce140 0x7f0e8413d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e82d9d700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415b2f0 0x7f0e8415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 -- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415b2f0 msgr2=0x7f0e8415b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415b2f0 0x7f0e8415b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 -- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e840ce140 msgr2=0x7f0e8413d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e840ce140 0x7f0e8413d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 -- 172.21.15.113:0/3038658195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7400b7f0 con 0x7f0e8415af10 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e81d9b700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e84158930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c4d0 tx=0x7f0e7800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800cea0 con 0x7f0e8415af10 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78011790 con 0x7f0e8415af10 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e84157890 con 0x7f0e8415af10 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e78011b40 con 0x7f0e8415af10 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.966+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e84157e50 con 0x7f0e8415af10 2024-01-16T13:52:53.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.967+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e84067f60 con 0x7f0e8415af10 2024-01-16T13:52:53.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.968+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0e78011ca0 con 0x7f0e8415af10 2024-01-16T13:52:53.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.968+0000 7f0e72ffd700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e6406b350 0x7f0e6406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:53.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.968+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e780895b0 con 0x7f0e8415af10 2024-01-16T13:52:53.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.968+0000 7f0e8259c700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e6406b350 0x7f0e6406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:53.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.969+0000 7f0e8259c700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e6406b350 0x7f0e6406d810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400b340 tx=0x7f0e7401d040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:53.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:53.973+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e780546d0 con 0x7f0e8415af10 2024-01-16T13:52:54.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.238+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f0e84072e00 con 0x7f0e8415af10 2024-01-16T13:52:54.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.243+0000 7f0e72ffd700 1 -- 172.21.15.113:0/3038658195 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0e78057cf0 con 0x7f0e8415af10 2024-01-16T13:52:54.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e6406b350 msgr2=0x7f0e6406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e6406b350 0x7f0e6406d810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400b340 tx=0x7f0e7401d040 comp rx=0 tx=0).stop 2024-01-16T13:52:54.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 msgr2=0x7f0e84158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e84158930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c4d0 tx=0x7f0e7800c7e0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 shutdown_connections 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f0e6406b350 0x7f0e6406d810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0e8415b2f0 0x7f0e8415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e840ce140 0x7f0e8413d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 --2- 172.21.15.113:0/3038658195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e8415af10 0x7f0e84158930 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 >> 172.21.15.113:0/3038658195 conn(0x7f0e840cd570 msgr2=0x7f0e8406ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 shutdown_connections 2024-01-16T13:52:54.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.245+0000 7f0e88d3e700 1 -- 172.21.15.113:0/3038658195 wait complete. 2024-01-16T13:52:54.239 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-16T13:52:54.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.380+0000 7f11a96a2700 1 Processor -- start 2024-01-16T13:52:54.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.380+0000 7f11a96a2700 1 -- start start 2024-01-16T13:52:54.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a96a2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11a415e0f0 0x7f11a40b9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a96a2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a406bc00 con 0x7f11a40b2b60 2024-01-16T13:52:54.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a96a2700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a415e4d0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a96a2700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a406aff0 con 0x7f11a415e0f0 2024-01-16T13:52:54.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a2ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11a415e0f0 0x7f11a40b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a27fc700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f11a40b2b60 0x7f11a40ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55966/0 (socket says 172.21.15.113:55966) 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a27fc700 1 -- 172.21.15.113:0/1563426826 learned_addr learned my addr 172.21.15.113:0/1563426826 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a2ffd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11a415e0f0 0x7f11a40b9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48782/0 (socket says 172.21.15.113:48782) 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a2ffd700 1 -- 172.21.15.113:0/1563426826 >> v1:172.21.15.124:6789/0 conn(0x7f11a40cf2a0 legacy=0x7f11a40d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a2ffd700 1 -- 172.21.15.113:0/1563426826 >> v1:172.21.15.113:6789/0 conn(0x7f11a40b2b60 legacy=0x7f11a40ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a2ffd700 1 -- 172.21.15.113:0/1563426826 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a406b2d0 con 0x7f11a415e0f0 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a1ffb700 1 -- 172.21.15.113:0/1563426826 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958541893 0 0) 0x7f11a406bc00 con 0x7f11a40b2b60 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a2ffd700 1 --2- 172.21.15.113:0/1563426826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a415e0f0 0x7f11a40b9700 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1194009840 tx=0x7f1194009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bf3ba9a4d2a8f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a1ffb700 1 -- 172.21.15.113:0/1563426826 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194004230 con 0x7f11a415e0f0 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a1ffb700 1 -- 172.21.15.113:0/1563426826 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194004390 con 0x7f11a415e0f0 2024-01-16T13:52:54.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.381+0000 7f11a1ffb700 1 -- 172.21.15.113:0/1563426826 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940036a0 con 0x7f11a415e0f0 2024-01-16T13:52:54.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 -- 172.21.15.113:0/1563426826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a415e0f0 msgr2=0x7f11a40b9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 --2- 172.21.15.113:0/1563426826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a415e0f0 0x7f11a40b9700 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1194009840 tx=0x7f1194009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:54.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 -- 172.21.15.113:0/1563426826 shutdown_connections 2024-01-16T13:52:54.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 --2- 172.21.15.113:0/1563426826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a415e0f0 0x7f11a40b9700 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 -- 172.21.15.113:0/1563426826 >> 172.21.15.113:0/1563426826 conn(0x7f11a40b5ce0 msgr2=0x7f11a40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:54.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 -- 172.21.15.113:0/1563426826 shutdown_connections 2024-01-16T13:52:54.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.382+0000 7f11a96a2700 1 -- 172.21.15.113:0/1563426826 wait complete. 2024-01-16T13:52:54.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.383+0000 7f11a96a2700 1 Processor -- start 2024-01-16T13:52:54.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.383+0000 7f11a96a2700 1 -- start start 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a96a2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f11a40b2b60 0x7f11a41548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a96a2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 0x7f11a40d3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a96a2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11a415e0f0 0x7f11a40d3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a96a2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40cdcb0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a96a2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41f3c60 con 0x7f11a415e0f0 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a96a2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40c9130 con 0x7f11a40b2b60 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 0x7f11a40d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 0x7f11a40d3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48786/0 (socket says 172.21.15.113:48786) 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 -- 172.21.15.113:0/2553768317 learned_addr learned my addr 172.21.15.113:0/2553768317 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a2ffd700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f11a40b2b60 0x7f11a41548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 -- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f11a40b2b60 msgr2=0x7f11a41548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f11a40b2b60 0x7f11a41548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 -- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11a415e0f0 msgr2=0x7f11a40d3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a37fe700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11a415e0f0 0x7f11a40d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.384+0000 7f11a27fc700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11a415e0f0 0x7f11a40d3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f11a27fc700 1 -- 172.21.15.113:0/2553768317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940094a0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f11a37fe700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11a415e0f0 0x7f11a40d3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f11a27fc700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 0x7f11a40d3110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f119800c520 tx=0x7f119800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198004c80 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198004e00 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a40c8520 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198007310 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.385+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a40c8ae0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.386+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a40c92e0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.386+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1198007470 con 0x7f11a40cf2a0 2024-01-16T13:52:54.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.387+0000 7f118b7fe700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f119006b350 0x7f119006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.387+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f11980895b0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.387+0000 7f11a2ffd700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f119006b350 0x7f119006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.387+0000 7f11a2ffd700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f119006b350 0x7f119006d810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1194009b90 tx=0x7f1194013040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:54.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.391+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11a40c92e0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.663+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f11a40c92a0 con 0x7f11a40cf2a0 2024-01-16T13:52:54.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.665+0000 7f118b7fe700 1 -- 172.21.15.113:0/2553768317 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1198054760 con 0x7f11a40cf2a0 2024-01-16T13:52:54.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f119006b350 msgr2=0x7f119006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f119006b350 0x7f119006d810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1194009b90 tx=0x7f1194013040 comp rx=0 tx=0).stop 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 msgr2=0x7f11a40d3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 0x7f11a40d3110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f119800c520 tx=0x7f119800c8e0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 shutdown_connections 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f119006b350 0x7f119006d810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11a415e0f0 0x7f11a40d3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f11a40b2b60 0x7f11a41548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 --2- 172.21.15.113:0/2553768317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11a40cf2a0 0x7f11a40d3110 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 >> 172.21.15.113:0/2553768317 conn(0x7f11a40b5ce0 msgr2=0x7f11a40b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 shutdown_connections 2024-01-16T13:52:54.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.667+0000 7f11a96a2700 1 -- 172.21.15.113:0/2553768317 wait complete. 2024-01-16T13:52:54.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-16T13:52:54.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.802+0000 7fa3684e6700 1 Processor -- start 2024-01-16T13:52:54.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.802+0000 7fa3684e6700 1 -- start start 2024-01-16T13:52:54.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa3684e6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa3601473a0 0x7fa36015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa3684e6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa36006dc20 0x7fa360073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa3684e6700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa36015eb00 con 0x7fa360147780 2024-01-16T13:52:54.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa3684e6700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36015ede0 con 0x7fa3601473a0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa3684e6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36015f0c0 con 0x7fa36006dc20 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa3601473a0 0x7fa36015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa3601473a0 0x7fa36015e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48798/0 (socket says 172.21.15.113:48798) 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 -- 172.21.15.113:0/3977350045 learned_addr learned my addr 172.21.15.113:0/3977350045 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366a83700 1 --2- 172.21.15.113:0/3977350045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa36006dc20 0x7fa360073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 -- 172.21.15.113:0/3977350045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa36006dc20 msgr2=0x7fa360073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 --2- 172.21.15.113:0/3977350045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa36006dc20 0x7fa360073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 -- 172.21.15.113:0/3977350045 >> v1:172.21.15.124:6789/0 conn(0x7fa360147780 legacy=0x7fa36006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 -- 172.21.15.113:0/3977350045 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36015f3a0 con 0x7fa3601473a0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa366282700 1 --2- 172.21.15.113:0/3977350045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3601473a0 0x7fa36015e4b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0098f0 tx=0x7fa35c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef65a1bab526f71b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa365280700 1 -- 172.21.15.113:0/3977350045 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c004230 con 0x7fa3601473a0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa365280700 1 -- 172.21.15.113:0/3977350045 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c004390 con 0x7fa3601473a0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.803+0000 7fa365280700 1 -- 172.21.15.113:0/3977350045 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0036a0 con 0x7fa3601473a0 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 -- 172.21.15.113:0/3977350045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3601473a0 msgr2=0x7fa36015e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 --2- 172.21.15.113:0/3977350045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3601473a0 0x7fa36015e4b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0098f0 tx=0x7fa35c009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:54.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 -- 172.21.15.113:0/3977350045 shutdown_connections 2024-01-16T13:52:54.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 --2- 172.21.15.113:0/3977350045 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa36006dc20 0x7fa360073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 --2- 172.21.15.113:0/3977350045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3601473a0 0x7fa36015e4b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 -- 172.21.15.113:0/3977350045 >> 172.21.15.113:0/3977350045 conn(0x7fa3600b3130 msgr2=0x7fa3600d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:54.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 -- 172.21.15.113:0/3977350045 shutdown_connections 2024-01-16T13:52:54.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.804+0000 7fa3684e6700 1 -- 172.21.15.113:0/3977350045 wait complete. 2024-01-16T13:52:54.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.805+0000 7fa3684e6700 1 Processor -- start 2024-01-16T13:52:54.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.805+0000 7fa3684e6700 1 -- start start 2024-01-16T13:52:54.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa3684e6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 0x7fa3600ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa3684e6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa3601473a0 0x7fa360144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa3684e6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa360147780 0x7fa360140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa3684e6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600ceee0 con 0x7fa36006dc20 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa3684e6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600cf1c0 con 0x7fa3601473a0 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa3684e6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3600cf4a0 con 0x7fa360147780 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 0x7fa3600ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 0x7fa3600ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48812/0 (socket says 172.21.15.113:48812) 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 -- 172.21.15.113:0/1409136667 learned_addr learned my addr 172.21.15.113:0/1409136667 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 -- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa360147780 msgr2=0x7fa360140e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa360147780 0x7fa360140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 -- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa3601473a0 msgr2=0x7fa360144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa365a81700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa3601473a0 0x7fa360144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa3601473a0 0x7fa360144f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 -- 172.21.15.113:0/1409136667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c009550 con 0x7fa36006dc20 2024-01-16T13:52:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.806+0000 7fa366282700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 0x7fa3600ce5b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009130 tx=0x7fa35c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.807+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c00b660 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.807+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c00b7e0 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.807+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c00bb90 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.807+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3600cf780 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.807+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3600ceaf0 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.807+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3600cecb0 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.808+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa35c021070 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.808+0000 7fa356ffd700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa34406b350 0x7fa34406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.808+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa35c08b180 con 0x7fa36006dc20 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.808+0000 7fa365a81700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa34406b350 0x7fa34406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.809+0000 7fa365a81700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa34406b350 0x7fa34406d810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa3600cd2c0 tx=0x7fa350008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:54.813+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa35c0563d0 con 0x7fa36006dc20 2024-01-16T13:52:55.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.085+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fa360067f60 con 0x7fa36006dc20 2024-01-16T13:52:55.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.086+0000 7fa356ffd700 1 -- 172.21.15.113:0/1409136667 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa35c00bcf0 con 0x7fa36006dc20 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa34406b350 msgr2=0x7fa34406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa34406b350 0x7fa34406d810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa3600cd2c0 tx=0x7fa350008040 comp rx=0 tx=0).stop 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 msgr2=0x7fa3600ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 0x7fa3600ce5b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009130 tx=0x7fa35c004750 comp rx=0 tx=0).stop 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 shutdown_connections 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa34406b350 0x7fa34406d810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa360147780 0x7fa360140e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36006dc20 0x7fa3600ce5b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 --2- 172.21.15.113:0/1409136667 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa3601473a0 0x7fa360144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 >> 172.21.15.113:0/1409136667 conn(0x7fa3600b3130 msgr2=0x7fa36015e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:55.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 shutdown_connections 2024-01-16T13:52:55.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.089+0000 7fa3684e6700 1 -- 172.21.15.113:0/1409136667 wait complete. 2024-01-16T13:52:55.083 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-16T13:52:55.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.222+0000 7f1e98fa5700 1 Processor -- start 2024-01-16T13:52:55.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.222+0000 7f1e98fa5700 1 -- start start 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e98fa5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1e94147780 0x7f1e9415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e98fa5700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9415eb00 con 0x7f1e9406dc20 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e98fa5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9415ede0 con 0x7f1e941473a0 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e98fa5700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9415f0c0 con 0x7f1e94147780 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e91d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1e94147780 0x7f1e9415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e92d9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1e9406dc20 0x7f1e94073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55970/0 (socket says 172.21.15.113:55970) 2024-01-16T13:52:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e92d9d700 1 -- 172.21.15.113:0/3643557911 learned_addr learned my addr 172.21.15.113:0/3643557911 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e91d9b700 1 -- 172.21.15.113:0/3643557911 >> v1:172.21.15.124:6789/0 conn(0x7f1e941473a0 legacy=0x7f1e9406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e91d9b700 1 -- 172.21.15.113:0/3643557911 >> v1:172.21.15.113:6789/0 conn(0x7f1e9406dc20 legacy=0x7f1e94073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e91d9b700 1 -- 172.21.15.113:0/3643557911 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9415f3a0 con 0x7f1e94147780 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e9159a700 1 -- 172.21.15.113:0/3643557911 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276522158 0 0) 0x7f1e9415eb00 con 0x7f1e9406dc20 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.223+0000 7f1e91d9b700 1 --2- 172.21.15.113:0/3643557911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e9415e4b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009840 tx=0x7f1e88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22525f6f92a64b3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e9159a700 1 -- 172.21.15.113:0/3643557911 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88011830 con 0x7f1e94147780 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e9159a700 1 -- 172.21.15.113:0/3643557911 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e88011990 con 0x7f1e94147780 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e9159a700 1 -- 172.21.15.113:0/3643557911 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88011d20 con 0x7f1e94147780 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 -- 172.21.15.113:0/3643557911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 msgr2=0x7f1e9415e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/3643557911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e9415e4b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009840 tx=0x7f1e88009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 -- 172.21.15.113:0/3643557911 shutdown_connections 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/3643557911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e9415e4b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 -- 172.21.15.113:0/3643557911 >> 172.21.15.113:0/3643557911 conn(0x7f1e940b3130 msgr2=0x7f1e940d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:55.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 -- 172.21.15.113:0/3643557911 shutdown_connections 2024-01-16T13:52:55.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.224+0000 7f1e98fa5700 1 -- 172.21.15.113:0/3643557911 wait complete. 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.226+0000 7f1e98fa5700 1 Processor -- start 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.226+0000 7f1e98fa5700 1 -- start start 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e98fa5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e9406dc20 0x7f1e940cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e98fa5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e941473a0 0x7f1e94135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e98fa5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e940aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e98fa5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c5ac0 con 0x7f1e94147780 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e98fa5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e941340c0 con 0x7f1e9406dc20 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e98fa5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c5ca0 con 0x7f1e941473a0 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e940aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e940aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48832/0 (socket says 172.21.15.113:48832) 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 -- 172.21.15.113:0/2176935113 learned_addr learned my addr 172.21.15.113:0/2176935113 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e9259c700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e9406dc20 0x7f1e940cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e91d9b700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e941473a0 0x7f1e94135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 -- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e941473a0 msgr2=0x7f1e94135fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e941473a0 0x7f1e94135fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 -- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e9406dc20 msgr2=0x7f1e940cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e9406dc20 0x7f1e940cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 -- 172.21.15.113:0/2176935113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880094a0 con 0x7f1e94147780 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e91d9b700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e941473a0 0x7f1e94135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e92d9d700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e940aff10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00cc70 tx=0x7f1e8c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c00ada0 con 0x7f1e94147780 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.227+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c00ce70 con 0x7f1e94147780 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.228+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0072f0 con 0x7f1e94147780 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.228+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e940c5f80 con 0x7f1e94147780 2024-01-16T13:52:55.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.228+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e940c6260 con 0x7f1e94147780 2024-01-16T13:52:55.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.228+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e94067f60 con 0x7f1e94147780 2024-01-16T13:52:55.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.229+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1e8c007450 con 0x7f1e94147780 2024-01-16T13:52:55.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.229+0000 7f1e82ffd700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e7406b350 0x7f1e7406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.229+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1e8c089430 con 0x7f1e94147780 2024-01-16T13:52:55.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.229+0000 7f1e9259c700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e7406b350 0x7f1e7406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.230+0000 7f1e9259c700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e7406b350 0x7f1e7406d810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e84000f60 tx=0x7f1e84009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.234+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e8c054560 con 0x7f1e94147780 2024-01-16T13:52:55.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.503+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f1e94134200 con 0x7f1e94147780 2024-01-16T13:52:55.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.504+0000 7f1e82ffd700 1 -- 172.21.15.113:0/2176935113 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1e8c018020 con 0x7f1e94147780 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e7406b350 msgr2=0x7f1e7406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e7406b350 0x7f1e7406d810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e84000f60 tx=0x7f1e84009040 comp rx=0 tx=0).stop 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 msgr2=0x7f1e940aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e940aff10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00cc70 tx=0x7f1e8c00a340 comp rx=0 tx=0).stop 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 shutdown_connections 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1e7406b350 0x7f1e7406d810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e94147780 0x7f1e940aff10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1e9406dc20 0x7f1e940cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 --2- 172.21.15.113:0/2176935113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1e941473a0 0x7f1e94135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 >> 172.21.15.113:0/2176935113 conn(0x7f1e940b3130 msgr2=0x7f1e9406f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:55.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 shutdown_connections 2024-01-16T13:52:55.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.507+0000 7f1e98fa5700 1 -- 172.21.15.113:0/2176935113 wait complete. 2024-01-16T13:52:55.501 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:55.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.639+0000 7f990cb22700 1 Processor -- start 2024-01-16T13:52:55.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.639+0000 7f990cb22700 1 -- start start 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990cb22700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f990815e080 0x7f99080cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990cb22700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f99080ce4d0 0x7f99080d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990cb22700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce910 0x7f99080d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990cb22700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990815e460 con 0x7f990815e080 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990cb22700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99080d0150 con 0x7f99080ce4d0 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990cb22700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080cee50 con 0x7f99080ce910 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990659c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f990815e080 0x7f99080cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990659c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f990815e080 0x7f99080cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48838/0 (socket says 172.21.15.113:48838) 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.640+0000 7f990659c700 1 -- 172.21.15.113:0/2410114326 learned_addr learned my addr 172.21.15.113:0/2410114326 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f9905d9b700 1 --2- 172.21.15.113:0/2410114326 >> v2:172.21.15.124:3300/0 conn(0x7f99080ce4d0 0x7f99080d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f9906d9d700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce910 0x7f99080d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990659c700 1 -- 172.21.15.113:0/2410114326 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce910 msgr2=0x7f99080d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:55.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990659c700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce910 0x7f99080d2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990659c700 1 -- 172.21.15.113:0/2410114326 >> v2:172.21.15.124:3300/0 conn(0x7f99080ce4d0 msgr2=0x7f99080d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990659c700 1 --2- 172.21.15.113:0/2410114326 >> v2:172.21.15.124:3300/0 conn(0x7f99080ce4d0 0x7f99080d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990659c700 1 -- 172.21.15.113:0/2410114326 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080cf130 con 0x7f990815e080 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f9906d9d700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce910 0x7f99080d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f9905d9b700 1 --2- 172.21.15.113:0/2410114326 >> v2:172.21.15.124:3300/0 conn(0x7f99080ce4d0 0x7f99080d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990659c700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f990815e080 0x7f99080cc2a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f99080ce290 tx=0x7f98fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a36fd87d7a39682 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990559a700 1 -- 172.21.15.113:0/2410114326 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00b870 con 0x7f990815e080 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990559a700 1 -- 172.21.15.113:0/2410114326 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc00b9d0 con 0x7f990815e080 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.641+0000 7f990559a700 1 -- 172.21.15.113:0/2410114326 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00bd60 con 0x7f990815e080 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 -- 172.21.15.113:0/2410114326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f990815e080 msgr2=0x7f99080cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f990815e080 0x7f99080cc2a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f99080ce290 tx=0x7f98fc009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 -- 172.21.15.113:0/2410114326 shutdown_connections 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f990815e080 0x7f99080cc2a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 --2- 172.21.15.113:0/2410114326 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce910 0x7f99080d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 --2- 172.21.15.113:0/2410114326 >> v2:172.21.15.124:3300/0 conn(0x7f99080ce4d0 0x7f99080d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 -- 172.21.15.113:0/2410114326 >> 172.21.15.113:0/2410114326 conn(0x7f9908073130 msgr2=0x7f990809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:55.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 -- 172.21.15.113:0/2410114326 shutdown_connections 2024-01-16T13:52:55.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.642+0000 7f990cb22700 1 -- 172.21.15.113:0/2410114326 wait complete. 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.643+0000 7f990cb22700 1 Processor -- start 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.643+0000 7f990cb22700 1 -- start start 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f990cb22700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce4d0 0x7f99080c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f990cb22700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 0x7f990815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f990cb22700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f990815e080 0x7f99080c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f990cb22700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080c5fb0 con 0x7f99080ce910 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f990cb22700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080c6290 con 0x7f990815e080 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f990cb22700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080c6570 con 0x7f99080ce4d0 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f9905d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 0x7f990815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f9905d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 0x7f990815c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48840/0 (socket says 172.21.15.113:48840) 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.644+0000 7f9905d9b700 1 -- 172.21.15.113:0/1100449668 learned_addr learned my addr 172.21.15.113:0/1100449668 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9905d9b700 1 -- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce4d0 msgr2=0x7f99080c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f990659c700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce4d0 0x7f99080c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9906d9d700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f990815e080 0x7f99080c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9905d9b700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce4d0 0x7f99080c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9905d9b700 1 -- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f990815e080 msgr2=0x7f99080c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9905d9b700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f990815e080 0x7f99080c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9905d9b700 1 -- 172.21.15.113:0/1100449668 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc009550 con 0x7f99080ce910 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f9905d9b700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 0x7f990815c750 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f98f800c750 tx=0x7f98f800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8011800 con 0x7f99080ce910 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f8011960 con 0x7f99080ce910 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99080c6850 con 0x7f99080ce910 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8011cd0 con 0x7f99080ce910 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.645+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99080c5ca0 con 0x7f99080ce910 2024-01-16T13:52:55.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.646+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99080c6d70 con 0x7f99080ce910 2024-01-16T13:52:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.646+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f98f8021350 con 0x7f99080ce910 2024-01-16T13:52:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.647+0000 7f98f6ffd700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f98e806b350 0x7f98e806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.647+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f98f80897b0 con 0x7f99080ce910 2024-01-16T13:52:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.647+0000 7f990659c700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f98e806b350 0x7f98e806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.647+0000 7f990659c700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f98e806b350 0x7f98e806d810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f99081f72b0 tx=0x7f98fc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.651+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99080c6d70 con 0x7f99080ce910 2024-01-16T13:52:55.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.842+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99080c6d70 con 0x7f99080ce910 2024-01-16T13:52:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.843+0000 7f98f6ffd700 1 -- 172.21.15.113:0/1100449668 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12893 (secure 0 0 0) 0x7f99080c6d70 con 0x7f99080ce910 2024-01-16T13:52:55.825 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:55.825 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":29,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:52.551215+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f98e806b350 msgr2=0x7f98e806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f98e806b350 0x7f98e806d810 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f99081f72b0 tx=0x7f98fc008030 comp rx=0 tx=0).stop 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 msgr2=0x7f990815c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 0x7f990815c750 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f98f800c750 tx=0x7f98f800cb10 comp rx=0 tx=0).stop 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 shutdown_connections 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f98e806b350 0x7f98e806d810 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f990815e080 0x7f99080c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99080ce4d0 0x7f99080c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 --2- 172.21.15.113:0/1100449668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99080ce910 0x7f990815c750 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 >> 172.21.15.113:0/1100449668 conn(0x7f9908073130 msgr2=0x7f990809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 shutdown_connections 2024-01-16T13:52:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.845+0000 7f990cb22700 1 -- 172.21.15.113:0/1100449668 wait complete. 2024-01-16T13:52:55.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:52:55.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:52:55.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.982+0000 7f91eb8ae700 1 Processor -- start 2024-01-16T13:52:55.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.983+0000 7f91eb8ae700 1 -- start start 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.983+0000 7f91eb8ae700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.983+0000 7f91eb8ae700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f91e406f0a0 0x7f91e40d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.983+0000 7f91eb8ae700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f91e40ceb20 0x7f91e406d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.983+0000 7f91eb8ae700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e4160700 con 0x7f91e406f0a0 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.983+0000 7f91eb8ae700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e415faf0 con 0x7f91e40ceb20 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91eb8ae700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e415fdd0 con 0x7f91e4159aa0 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f91e406f0a0 0x7f91e40d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f91e406f0a0 0x7f91e40d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48842/0 (socket says 172.21.15.113:48842) 2024-01-16T13:52:55.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 -- 172.21.15.113:0/2371274019 learned_addr learned my addr 172.21.15.113:0/2371274019 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 -- 172.21.15.113:0/2371274019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 msgr2=0x7f91e40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2371274019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 -- 172.21.15.113:0/2371274019 >> v2:172.21.15.124:3300/0 conn(0x7f91e40ceb20 msgr2=0x7f91e406d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e9e4b700 1 --2- 172.21.15.113:0/2371274019 >> v2:172.21.15.124:3300/0 conn(0x7f91e40ceb20 0x7f91e406d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2371274019 >> v2:172.21.15.124:3300/0 conn(0x7f91e40ceb20 0x7f91e406d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 -- 172.21.15.113:0/2371274019 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e41600b0 con 0x7f91e406f0a0 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2371274019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e40d0840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f91e40afb10 tx=0x7f91e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b1cefb7850de14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91d7fff700 1 -- 172.21.15.113:0/2371274019 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e000b870 con 0x7f91e406f0a0 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91d7fff700 1 -- 172.21.15.113:0/2371274019 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e000b9d0 con 0x7f91e406f0a0 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.984+0000 7f91d7fff700 1 -- 172.21.15.113:0/2371274019 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e000bd60 con 0x7f91e406f0a0 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2371274019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 msgr2=0x7f91e40d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2371274019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e40d0840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f91e40afb10 tx=0x7f91e0009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2371274019 shutdown_connections 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2371274019 >> v2:172.21.15.124:3300/0 conn(0x7f91e40ceb20 0x7f91e406d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2371274019 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2371274019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e40d0840 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2371274019 >> 172.21.15.113:0/2371274019 conn(0x7f91e40743b0 msgr2=0x7f91e406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:55.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2371274019 shutdown_connections 2024-01-16T13:52:55.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.985+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2371274019 wait complete. 2024-01-16T13:52:55.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.986+0000 7f91eb8ae700 1 Processor -- start 2024-01-16T13:52:55.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.986+0000 7f91eb8ae700 1 -- start start 2024-01-16T13:52:55.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91eb8ae700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e415daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91eb8ae700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e40ceb20 0x7f91e4143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91eb8ae700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e4152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91eb8ae700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e4157e80 con 0x7f91e406f0a0 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91eb8ae700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e415ee70 con 0x7f91e40ceb20 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91eb8ae700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e415dfe0 con 0x7f91e4159aa0 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e415daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e415daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48858/0 (socket says 172.21.15.113:48858) 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 -- 172.21.15.113:0/2983804104 learned_addr learned my addr 172.21.15.113:0/2983804104 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e9e4b700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e4152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e40ceb20 0x7f91e4143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 -- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 msgr2=0x7f91e4152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e4152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 -- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e40ceb20 msgr2=0x7f91e4143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e40ceb20 0x7f91e4143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 -- 172.21.15.113:0/2983804104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0009550 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e9e4b700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e4152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e40ceb20 0x7f91e4143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91e964a700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e415daa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b4a0 tx=0x7f91d800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.987+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8011830 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.988+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d8011990 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.988+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8011d40 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.988+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e415e260 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.988+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e415e820 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.989+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e40ce760 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.989+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f91d801a400 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.989+0000 7f91d5ffb700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f91c806b380 0x7f91c806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.990+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f91c806b380 0x7f91c806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.990+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91d8089800 con 0x7f91e406f0a0 2024-01-16T13:52:55.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.990+0000 7f91e8e49700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f91c806b380 0x7f91c806d840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f91e0000c00 tx=0x7f91e0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:55.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:55.994+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91d80549d0 con 0x7f91e406f0a0 2024-01-16T13:52:56.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.183+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91e4069020 con 0x7f91e406f0a0 2024-01-16T13:52:56.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.184+0000 7f91d5ffb700 1 -- 172.21.15.113:0/2983804104 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91d8057ff0 con 0x7f91e406f0a0 2024-01-16T13:52:56.166 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:52:56.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f91c806b380 msgr2=0x7f91c806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f91c806b380 0x7f91c806d840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f91e0000c00 tx=0x7f91e0008030 comp rx=0 tx=0).stop 2024-01-16T13:52:56.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 msgr2=0x7f91e415daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e415daa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b4a0 tx=0x7f91d800b860 comp rx=0 tx=0).stop 2024-01-16T13:52:56.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 shutdown_connections 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f91c806b380 0x7f91c806d840 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e4159aa0 0x7f91e4152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91e406f0a0 0x7f91e415daa0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 --2- 172.21.15.113:0/2983804104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e40ceb20 0x7f91e4143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 >> 172.21.15.113:0/2983804104 conn(0x7f91e40743b0 msgr2=0x7f91e40d00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 shutdown_connections 2024-01-16T13:52:56.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.186+0000 7f91eb8ae700 1 -- 172.21.15.113:0/2983804104 wait complete. 2024-01-16T13:52:56.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T13:52:56.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.319+0000 7feb6885b700 1 Processor -- start 2024-01-16T13:52:56.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.319+0000 7feb6885b700 1 -- start start 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6885b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feb640ce4d0 0x7feb640d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6885b700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb640d0150 con 0x7feb640ce910 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6885b700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb640cee50 con 0x7feb6415e080 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6885b700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6415e460 con 0x7feb640ce4d0 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb61d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7feb640ce4d0 0x7feb640d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb62d9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7feb640ce910 0x7feb640d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55988/0 (socket says 172.21.15.113:55988) 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb62d9d700 1 -- 172.21.15.113:0/1992631424 learned_addr learned my addr 172.21.15.113:0/1992631424 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:56.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb61d9b700 1 -- 172.21.15.113:0/1992631424 >> v1:172.21.15.124:6789/0 conn(0x7feb6415e080 legacy=0x7feb640cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb61d9b700 1 -- 172.21.15.113:0/1992631424 >> v1:172.21.15.113:6789/0 conn(0x7feb640ce910 legacy=0x7feb640d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6159a700 1 -- 172.21.15.113:0/1992631424 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433917686 0 0) 0x7feb640d0150 con 0x7feb640ce910 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb61d9b700 1 -- 172.21.15.113:0/1992631424 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb640cf130 con 0x7feb640ce4d0 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb61d9b700 1 --2- 172.21.15.113:0/1992631424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb640d1be0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7feb58009840 tx=0x7feb58009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6009d4d24ac0a575 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6159a700 1 -- 172.21.15.113:0/1992631424 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58011830 con 0x7feb640ce4d0 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6159a700 1 -- 172.21.15.113:0/1992631424 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb58011990 con 0x7feb640ce4d0 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.320+0000 7feb6159a700 1 -- 172.21.15.113:0/1992631424 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58011d20 con 0x7feb640ce4d0 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 -- 172.21.15.113:0/1992631424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 msgr2=0x7feb640d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 --2- 172.21.15.113:0/1992631424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb640d1be0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7feb58009840 tx=0x7feb58009b50 comp rx=0 tx=0).stop 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 -- 172.21.15.113:0/1992631424 shutdown_connections 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 --2- 172.21.15.113:0/1992631424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb640d1be0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 -- 172.21.15.113:0/1992631424 >> 172.21.15.113:0/1992631424 conn(0x7feb64073130 msgr2=0x7feb6409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 -- 172.21.15.113:0/1992631424 shutdown_connections 2024-01-16T13:52:56.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.321+0000 7feb6885b700 1 -- 172.21.15.113:0/1992631424 wait complete. 2024-01-16T13:52:56.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.322+0000 7feb6885b700 1 Processor -- start 2024-01-16T13:52:56.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.322+0000 7feb6885b700 1 -- start start 2024-01-16T13:52:56.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6885b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb6414bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6885b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feb640ce910 0x7feb6414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6885b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feb6415e080 0x7feb640c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6885b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6415c860 con 0x7feb640ce4d0 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6885b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb640b5b40 con 0x7feb640ce910 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6885b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb640b5e20 con 0x7feb6415e080 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb6414bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb6414bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48864/0 (socket says 172.21.15.113:48864) 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 -- 172.21.15.113:0/1270053286 learned_addr learned my addr 172.21.15.113:0/1270053286 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb61d9b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feb640ce910 0x7feb6414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb62d9d700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feb6415e080 0x7feb640c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 -- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feb6415e080 msgr2=0x7feb640c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feb6415e080 0x7feb640c9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 -- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feb640ce910 msgr2=0x7feb6414c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feb640ce910 0x7feb6414c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb6259c700 1 -- 172.21.15.113:0/1270053286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580094a0 con 0x7feb640ce4d0 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.323+0000 7feb62d9d700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feb6415e080 0x7feb640c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:56.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.324+0000 7feb6259c700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb6414bf60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feb5400c450 tx=0x7feb5400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.324+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540052d0 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.324+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb54005430 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.324+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb640b6100 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.324+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb540038c0 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.324+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb640ca2a0 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.325+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64067f60 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.325+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feb54003a20 con 0x7feb640ce4d0 2024-01-16T13:52:56.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.325+0000 7feb52ffd700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feb4406b350 0x7feb4406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.326+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb54088ac0 con 0x7feb640ce4d0 2024-01-16T13:52:56.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.326+0000 7feb61d9b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feb4406b350 0x7feb4406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.326+0000 7feb61d9b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feb4406b350 0x7feb4406d810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feb640b59a0 tx=0x7feb5801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:56.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.330+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb54053d10 con 0x7feb640ce4d0 2024-01-16T13:52:56.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.526+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feb640d19c0 con 0x7feb640ce4d0 2024-01-16T13:52:56.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.526+0000 7feb52ffd700 1 -- 172.21.15.113:0/1270053286 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7feb54057330 con 0x7feb640ce4d0 2024-01-16T13:52:56.508 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:56.508 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-16T13:52:56.509 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:52:56.509 INFO:teuthology.orchestra.run.smithi113.stdout: "poolnum": 1, 2024-01-16T13:52:56.509 INFO:teuthology.orchestra.run.smithi113.stdout: "poolname": ".mgr" 2024-01-16T13:52:56.509 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:52:56.509 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.528+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feb4406b350 msgr2=0x7feb4406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.528+0000 7feb6885b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feb4406b350 0x7feb4406d810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feb640b59a0 tx=0x7feb5801c040 comp rx=0 tx=0).stop 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.528+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 msgr2=0x7feb6414bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.528+0000 7feb6885b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb6414bf60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feb5400c450 tx=0x7feb5400c810 comp rx=0 tx=0).stop 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 shutdown_connections 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feb4406b350 0x7feb4406d810 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feb6415e080 0x7feb640c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb640ce4d0 0x7feb6414bf60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 --2- 172.21.15.113:0/1270053286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feb640ce910 0x7feb6414c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 >> 172.21.15.113:0/1270053286 conn(0x7feb64073130 msgr2=0x7feb640d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 shutdown_connections 2024-01-16T13:52:56.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.529+0000 7feb6885b700 1 -- 172.21.15.113:0/1270053286 wait complete. 2024-01-16T13:52:56.523 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T13:52:56.523 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T13:52:56.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.667+0000 7f4dc3c0a700 1 Processor -- start 2024-01-16T13:52:56.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.667+0000 7f4dc3c0a700 1 -- start start 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc3c0a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc3c0a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4dbc162820 0x7f4dbc06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc3c0a700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc06bed0 con 0x7f4dbc15df20 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc3c0a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc06cfe0 con 0x7f4dbc162820 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc3c0a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc06bba0 con 0x7f4dbc1622b0 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc19a6700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f4dbc15df20 0x7f4dbc161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:55990/0 (socket says 172.21.15.113:55990) 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc19a6700 1 -- 172.21.15.113:0/301630797 learned_addr learned my addr 172.21.15.113:0/301630797 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc21a7700 1 --2- 172.21.15.113:0/301630797 >> v2:172.21.15.113:3300/0 conn(0x7f4dbc162820 0x7f4dbc06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc11a5700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc21a7700 1 -- 172.21.15.113:0/301630797 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 msgr2=0x7f4dbc06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc21a7700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc06c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.668+0000 7f4dc21a7700 1 -- 172.21.15.113:0/301630797 >> v1:172.21.15.113:6789/0 conn(0x7f4dbc15df20 legacy=0x7f4dbc161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc21a7700 1 -- 172.21.15.113:0/301630797 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc070060 con 0x7f4dbc162820 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc09a4700 1 -- 172.21.15.113:0/301630797 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286931198 0 0) 0x7f4dbc06bed0 con 0x7f4dbc15df20 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc11a5700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc06c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc21a7700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc162820 0x7f4dbc06caa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009d90 tx=0x7f4db400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b808ba2929cbd676 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc09a4700 1 -- 172.21.15.113:0/301630797 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db400e070 con 0x7f4dbc162820 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc09a4700 1 -- 172.21.15.113:0/301630797 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db400bbd0 con 0x7f4dbc162820 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc09a4700 1 -- 172.21.15.113:0/301630797 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4012430 con 0x7f4dbc162820 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/301630797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc162820 msgr2=0x7f4dbc06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc162820 0x7f4dbc06caa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009d90 tx=0x7f4db400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/301630797 shutdown_connections 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc162820 0x7f4dbc06caa0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/301630797 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc06c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/301630797 >> 172.21.15.113:0/301630797 conn(0x7f4dbc06dc80 msgr2=0x7f4dbc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/301630797 shutdown_connections 2024-01-16T13:52:56.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.669+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/301630797 wait complete. 2024-01-16T13:52:56.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 Processor -- start 2024-01-16T13:52:56.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 -- start start 2024-01-16T13:52:56.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 0x7f4dbc149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4dbc162820 0x7f4dbc14bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc1e8660 con 0x7f4dbc15df20 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc14a240 con 0x7f4dbc162820 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc3c0a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0b7a20 con 0x7f4dbc1622b0 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc19a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 0x7f4dbc149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc19a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 0x7f4dbc149d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48890/0 (socket says 172.21.15.113:48890) 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.671+0000 7f4dc19a6700 1 -- 172.21.15.113:0/1821305575 learned_addr learned my addr 172.21.15.113:0/1821305575 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc11a5700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc21a7700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4dbc162820 0x7f4dbc14bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc19a6700 1 -- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 msgr2=0x7f4dbc0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc19a6700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc0cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc19a6700 1 -- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4dbc162820 msgr2=0x7f4dbc14bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc19a6700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4dbc162820 0x7f4dbc14bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:56.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc19a6700 1 -- 172.21.15.113:0/1821305575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4009a70 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc21a7700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4dbc162820 0x7f4dbc14bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc19a6700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 0x7f4dbc149d00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c580 tx=0x7f4dac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac004c80 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac004e00 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc1e7d30 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac007310 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.672+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc1e82f0 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.673+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc067f60 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.673+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4dac007470 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.674+0000 7f4db27fc700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4da006b350 0x7f4da006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.674+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4dac089500 con 0x7f4dbc15df20 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.674+0000 7f4dc11a5700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4da006b350 0x7f4da006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:56.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.674+0000 7f4dc11a5700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4da006b350 0x7f4da006d810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc1ea710 tx=0x7f4db8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:56.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.678+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4dac0546c0 con 0x7f4dbc15df20 2024-01-16T13:52:56.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:56.872+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4dbc0d2030 con 0x7f4dbc15df20 2024-01-16T13:52:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.534+0000 7f4db27fc700 1 -- 172.21.15.113:0/1821305575 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4dac057ce0 con 0x7f4dbc15df20 2024-01-16T13:52:57.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4da006b350 msgr2=0x7f4da006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4da006b350 0x7f4da006d810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc1ea710 tx=0x7f4db8008040 comp rx=0 tx=0).stop 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 msgr2=0x7f4dbc149d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 0x7f4dbc149d00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c580 tx=0x7f4dac00c890 comp rx=0 tx=0).stop 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 shutdown_connections 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f4da006b350 0x7f4da006d810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4dbc162820 0x7f4dbc14bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4dbc15df20 0x7f4dbc149d00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 --2- 172.21.15.113:0/1821305575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4dbc1622b0 0x7f4dbc0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.537+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 >> 172.21.15.113:0/1821305575 conn(0x7f4dbc06dc80 msgr2=0x7f4dbc0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:57.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.538+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 shutdown_connections 2024-01-16T13:52:57.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.538+0000 7f4dc3c0a700 1 -- 172.21.15.113:0/1821305575 wait complete. 2024-01-16T13:52:57.521 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_metadata' created 2024-01-16T13:52:57.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T13:52:57.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.689+0000 7f5223f2f700 1 Processor -- start 2024-01-16T13:52:57.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.689+0000 7f5223f2f700 1 -- start start 2024-01-16T13:52:57.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f5223f2f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f521c0746a0 0x7f521c168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:57.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f5223f2f700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c15e4a0 con 0x7f521c0afb80 2024-01-16T13:52:57.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f5223f2f700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c06aff0 con 0x7f521c15e0c0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f5223f2f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c0b0010 con 0x7f521c0746a0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52224cc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f521c0746a0 0x7f521c168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52214ca700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f521c0afb80 0x7f521c073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56000/0 (socket says 172.21.15.113:56000) 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52224cc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f521c0746a0 0x7f521c168fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48898/0 (socket says 172.21.15.113:48898) 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52214ca700 1 -- 172.21.15.113:0/482443969 learned_addr learned my addr 172.21.15.113:0/482443969 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52224cc700 1 -- 172.21.15.113:0/482443969 >> v1:172.21.15.124:6789/0 conn(0x7f521c15e0c0 legacy=0x7f521c0b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52224cc700 1 -- 172.21.15.113:0/482443969 >> v1:172.21.15.113:6789/0 conn(0x7f521c0afb80 legacy=0x7f521c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52224cc700 1 -- 172.21.15.113:0/482443969 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c06b2d0 con 0x7f521c0746a0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f5220cc9700 1 -- 172.21.15.113:0/482443969 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547998448 0 0) 0x7f521c15e4a0 con 0x7f521c0afb80 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.690+0000 7f52224cc700 1 --2- 172.21.15.113:0/482443969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0746a0 0x7f521c168fa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5214009dc0 tx=0x7f521400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1cc03c96b09030 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5220cc9700 1 -- 172.21.15.113:0/482443969 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400e070 con 0x7f521c0746a0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5220cc9700 1 -- 172.21.15.113:0/482443969 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521400bb50 con 0x7f521c0746a0 2024-01-16T13:52:57.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5220cc9700 1 -- 172.21.15.113:0/482443969 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214003a20 con 0x7f521c0746a0 2024-01-16T13:52:57.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 -- 172.21.15.113:0/482443969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0746a0 msgr2=0x7f521c168fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:57.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 --2- 172.21.15.113:0/482443969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0746a0 0x7f521c168fa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5214009dc0 tx=0x7f521400b330 comp rx=0 tx=0).stop 2024-01-16T13:52:57.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 -- 172.21.15.113:0/482443969 shutdown_connections 2024-01-16T13:52:57.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 --2- 172.21.15.113:0/482443969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0746a0 0x7f521c168fa0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 -- 172.21.15.113:0/482443969 >> 172.21.15.113:0/482443969 conn(0x7f521c0d0c00 msgr2=0x7f521c0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:57.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 -- 172.21.15.113:0/482443969 shutdown_connections 2024-01-16T13:52:57.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.691+0000 7f5223f2f700 1 -- 172.21.15.113:0/482443969 wait complete. 2024-01-16T13:52:57.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.692+0000 7f5223f2f700 1 Processor -- start 2024-01-16T13:52:57.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.692+0000 7f5223f2f700 1 -- start start 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5223f2f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f521c0746a0 0x7f521c1478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5223f2f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 0x7f521c159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5223f2f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f521c15e0c0 0x7f521c1596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5223f2f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0cd190 con 0x7f521c0afb80 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5223f2f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0cd470 con 0x7f521c0746a0 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5223f2f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0cd750 con 0x7f521c15e0c0 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 0x7f521c159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:57.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 0x7f521c159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48910/0 (socket says 172.21.15.113:48910) 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 -- 172.21.15.113:0/1845266156 learned_addr learned my addr 172.21.15.113:0/1845266156 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 -- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f521c15e0c0 msgr2=0x7f521c1596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f521c15e0c0 0x7f521c1596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 -- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f521c0746a0 msgr2=0x7f521c1478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f5221ccb700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f521c0746a0 0x7f521c1478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f521c0746a0 0x7f521c1478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.693+0000 7f52214ca700 1 -- 172.21.15.113:0/1845266156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214009a70 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.694+0000 7f52214ca700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 0x7f521c159180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f520c0011e0 tx=0x7f520c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.694+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c004c80 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.694+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c004e00 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.694+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c0cda30 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.694+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c007310 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.694+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c0ce230 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.695+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c0cdf50 con 0x7f521c0afb80 2024-01-16T13:52:57.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.695+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f520c007470 con 0x7f521c0afb80 2024-01-16T13:52:57.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.696+0000 7f52127fc700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f520006b350 0x7f520006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:57.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.696+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f520c089790 con 0x7f521c0afb80 2024-01-16T13:52:57.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.696+0000 7f5221ccb700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f520006b350 0x7f520006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:57.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.696+0000 7f5221ccb700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f520006b350 0x7f520006d810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f521800a7d0 tx=0x7f5218008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:57.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.700+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f521c0cdf50 con 0x7f521c0afb80 2024-01-16T13:52:57.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-01-16T13:52:57.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-01-16T13:52:57.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-01-16T13:52:57.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:57.890+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f521c0cdf50 con 0x7f521c0afb80 2024-01-16T13:52:58.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.541+0000 7f52127fc700 1 -- 172.21.15.113:0/1845266156 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7f521c0cdf50 con 0x7f521c0afb80 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f520006b350 msgr2=0x7f520006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f520006b350 0x7f520006d810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f521800a7d0 tx=0x7f5218008040 comp rx=0 tx=0).stop 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 msgr2=0x7f521c159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 0x7f521c159180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f520c0011e0 tx=0x7f520c00cae0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 shutdown_connections 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f520006b350 0x7f520006d810 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f521c15e0c0 0x7f521c1596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f521c0746a0 0x7f521c1478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 --2- 172.21.15.113:0/1845266156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f521c0afb80 0x7f521c159180 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f520c0011e0 tx=0x7f520c00cae0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.544+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 >> 172.21.15.113:0/1845266156 conn(0x7f521c0d0c00 msgr2=0x7f521c06e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.545+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 shutdown_connections 2024-01-16T13:52:58.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.545+0000 7f5223f2f700 1 -- 172.21.15.113:0/1845266156 wait complete. 2024-01-16T13:52:58.529 INFO:teuthology.orchestra.run.smithi113.stderr:pool 'cephfs_data' created 2024-01-16T13:52:58.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T13:52:58.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.693+0000 7ffa26b71700 1 Processor -- start 2024-01-16T13:52:58.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.693+0000 7ffa26b71700 1 -- start start 2024-01-16T13:52:58.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa26b71700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffa2015b240 0x7ffa2006e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:58.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa26b71700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa2015b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa26b71700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2006eeb0 con 0x7ffa2015bb60 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa26b71700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2006f190 con 0x7ffa2015b240 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa26b71700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2006f470 con 0x7ffa2006e9a0 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffa2015b240 0x7ffa2006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffa2015b240 0x7ffa2006e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48912/0 (socket says 172.21.15.113:48912) 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 -- 172.21.15.113:0/1174061026 learned_addr learned my addr 172.21.15.113:0/1174061026 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2510e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ffa2015bb60 0x7ffa2006b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56006/0 (socket says 172.21.15.113:56006) 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa1ffff700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa2015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 -- 172.21.15.113:0/1174061026 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 msgr2=0x7ffa2015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa2015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 -- 172.21.15.113:0/1174061026 >> v1:172.21.15.113:6789/0 conn(0x7ffa2015bb60 legacy=0x7ffa2006b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 -- 172.21.15.113:0/1174061026 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2006f750 con 0x7ffa2015b240 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa1f7fe700 1 -- 172.21.15.113:0/1174061026 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757272512 0 0) 0x7ffa2006eeb0 con 0x7ffa2015bb60 2024-01-16T13:52:58.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa1ffff700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa2015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa2490d700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa2006e460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffa140098f0 tx=0x7ffa14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56848f8957c63997 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa1f7fe700 1 -- 172.21.15.113:0/1174061026 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1400b8a0 con 0x7ffa2015b240 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa1f7fe700 1 -- 172.21.15.113:0/1174061026 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1400ba00 con 0x7ffa2015b240 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.694+0000 7ffa1f7fe700 1 -- 172.21.15.113:0/1174061026 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1400bd90 con 0x7ffa2015b240 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 -- 172.21.15.113:0/1174061026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 msgr2=0x7ffa2006e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa2006e460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffa140098f0 tx=0x7ffa14009c00 comp rx=0 tx=0).stop 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 -- 172.21.15.113:0/1174061026 shutdown_connections 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa2006e460 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 --2- 172.21.15.113:0/1174061026 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa2015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 -- 172.21.15.113:0/1174061026 >> 172.21.15.113:0/1174061026 conn(0x7ffa200d0c00 msgr2=0x7ffa200d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 -- 172.21.15.113:0/1174061026 shutdown_connections 2024-01-16T13:52:58.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.695+0000 7ffa26b71700 1 -- 172.21.15.113:0/1174061026 wait complete. 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.696+0000 7ffa26b71700 1 Processor -- start 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.696+0000 7ffa26b71700 1 -- start start 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa26b71700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa201529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa26b71700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa200cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa26b71700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa2015bb60 0x7ffa201566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa26b71700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20152f20 con 0x7ffa2015b240 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa26b71700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200cdd90 con 0x7ffa2015bb60 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa26b71700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200ce070 con 0x7ffa2006e9a0 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa200cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:58.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa200cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48926/0 (socket says 172.21.15.113:48926) 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 -- 172.21.15.113:0/3254946847 learned_addr learned my addr 172.21.15.113:0/3254946847 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa2510e700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa2015bb60 0x7ffa201566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa2490d700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa201529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 -- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 msgr2=0x7ffa201529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa201529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 -- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa2015bb60 msgr2=0x7ffa201566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa2015bb60 0x7ffa201566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.697+0000 7ffa1ffff700 1 -- 172.21.15.113:0/3254946847 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14009550 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.698+0000 7ffa1ffff700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa200cd3b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00c7f0 tx=0x7ffa0c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.698+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c011800 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.698+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c011960 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.698+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa200ce350 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.698+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c011d10 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.698+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa200ce910 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.699+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20069020 con 0x7ffa2015b240 2024-01-16T13:52:58.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.699+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffa0c018360 con 0x7ffa2015b240 2024-01-16T13:52:58.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.700+0000 7ffa1d7fa700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa0406b350 0x7ffa0406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:58.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.700+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ffa0c0898f0 con 0x7ffa2015b240 2024-01-16T13:52:58.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.700+0000 7ffa2490d700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa0406b350 0x7ffa0406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:58.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.700+0000 7ffa2490d700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa0406b350 0x7ffa0406d810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009c40 tx=0x7ffa14008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:58.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.705+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa0c054850 con 0x7ffa2015b240 2024-01-16T13:52:58.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:58.940+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ffa200d0730 con 0x7ffa2015b240 2024-01-16T13:52:59.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.542+0000 7ffa1d7fa700 1 -- 172.21.15.113:0/3254946847 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7ffa0c01f0c0 con 0x7ffa2015b240 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.544+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa0406b350 msgr2=0x7ffa0406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.544+0000 7ffa26b71700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa0406b350 0x7ffa0406d810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009c40 tx=0x7ffa14008030 comp rx=0 tx=0).stop 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.544+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 msgr2=0x7ffa200cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.544+0000 7ffa26b71700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa200cd3b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00c7f0 tx=0x7ffa0c00cb00 comp rx=0 tx=0).stop 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 shutdown_connections 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa0406b350 0x7ffa0406d810 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa2015bb60 0x7ffa201566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa2006e9a0 0x7ffa201529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 --2- 172.21.15.113:0/3254946847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa2015b240 0x7ffa200cd3b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 >> 172.21.15.113:0/3254946847 conn(0x7ffa200d0c00 msgr2=0x7ffa2006be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 shutdown_connections 2024-01-16T13:52:59.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.545+0000 7ffa26b71700 1 -- 172.21.15.113:0/3254946847 wait complete. 2024-01-16T13:52:59.528 INFO:teuthology.orchestra.run.smithi113.stderr:new fs with metadata pool 4 and data pool 5 2024-01-16T13:52:59.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:52:59.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.695+0000 7f63cb983700 1 Processor -- start 2024-01-16T13:52:59.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.695+0000 7f63cb983700 1 -- start start 2024-01-16T13:52:59.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63cb983700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63c415af10 0x7f63c415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63cb983700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63c40ce140 0x7f63c415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63cb983700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63cb983700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c41414c0 con 0x7f63c415af10 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63cb983700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c40ce9f0 con 0x7f63c40ce140 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63cb983700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40cecd0 con 0x7f63c415f460 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63c415af10 0x7f63c415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63c415af10 0x7f63c415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48930/0 (socket says 172.21.15.113:48930) 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 -- 172.21.15.113:0/87676781 learned_addr learned my addr 172.21.15.113:0/87676781 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 -- 172.21.15.113:0/87676781 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 msgr2=0x7f63c415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c8f1e700 1 --2- 172.21.15.113:0/87676781 >> v2:172.21.15.124:3300/0 conn(0x7f63c40ce140 0x7f63c415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c9f20700 1 --2- 172.21.15.113:0/87676781 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 --2- 172.21.15.113:0/87676781 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c415b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 -- 172.21.15.113:0/87676781 >> v2:172.21.15.124:3300/0 conn(0x7f63c40ce140 msgr2=0x7f63c415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:52:59.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 --2- 172.21.15.113:0/87676781 >> v2:172.21.15.124:3300/0 conn(0x7f63c40ce140 0x7f63c415ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 -- 172.21.15.113:0/87676781 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c406b120 con 0x7f63c415af10 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.696+0000 7f63c971f700 1 --2- 172.21.15.113:0/87676781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c415e9e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f63c406d090 tx=0x7f63c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8834deb66d7c936 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63bbfff700 1 -- 172.21.15.113:0/87676781 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000b870 con 0x7f63c415af10 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63bbfff700 1 -- 172.21.15.113:0/87676781 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c000b9d0 con 0x7f63c415af10 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63bbfff700 1 -- 172.21.15.113:0/87676781 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000bd60 con 0x7f63c415af10 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 -- 172.21.15.113:0/87676781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 msgr2=0x7f63c415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 --2- 172.21.15.113:0/87676781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c415e9e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f63c406d090 tx=0x7f63c0009bd0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 -- 172.21.15.113:0/87676781 shutdown_connections 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 --2- 172.21.15.113:0/87676781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c415e9e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 --2- 172.21.15.113:0/87676781 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 --2- 172.21.15.113:0/87676781 >> v2:172.21.15.124:3300/0 conn(0x7f63c40ce140 0x7f63c415ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 -- 172.21.15.113:0/87676781 >> 172.21.15.113:0/87676781 conn(0x7f63c40cd570 msgr2=0x7f63c40cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 -- 172.21.15.113:0/87676781 shutdown_connections 2024-01-16T13:52:59.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.697+0000 7f63cb983700 1 -- 172.21.15.113:0/87676781 wait complete. 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.698+0000 7f63cb983700 1 Processor -- start 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.698+0000 7f63cb983700 1 -- start start 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63cb983700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63c40ce140 0x7f63c4074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63cb983700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c41f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63cb983700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c41f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63cb983700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c41403c0 con 0x7f63c415af10 2024-01-16T13:52:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63cb983700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c41f15e0 con 0x7f63c40ce140 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63cb983700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c413f740 con 0x7f63c415f460 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63c8f1e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c41f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63c8f1e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c41f0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48940/0 (socket says 172.21.15.113:48940) 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63c8f1e700 1 -- 172.21.15.113:0/2340139568 learned_addr learned my addr 172.21.15.113:0/2340139568 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63c9f20700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c41f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.699+0000 7f63c971f700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63c40ce140 0x7f63c4074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c8f1e700 1 -- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 msgr2=0x7f63c41f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c8f1e700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c41f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c8f1e700 1 -- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63c40ce140 msgr2=0x7f63c4074200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c8f1e700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63c40ce140 0x7f63c4074200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c8f1e700 1 -- 172.21.15.113:0/2340139568 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0009550 con 0x7f63c415af10 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c971f700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63c40ce140 0x7f63c4074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63c8f1e700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c41f0b60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c820 tx=0x7f63b400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4011800 con 0x7f63c415af10 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4011960 con 0x7f63c415af10 2024-01-16T13:52:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c413fa20 con 0x7f63c415af10 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4011cd0 con 0x7f63c415af10 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.700+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c413ffe0 con 0x7f63c415af10 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.701+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4140500 con 0x7f63c415af10 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.701+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f63b4022350 con 0x7f63c415af10 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.702+0000 7f63b9ffb700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f63a806b350 0x7f63a806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.702+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63b40899f0 con 0x7f63c415af10 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.702+0000 7f63c971f700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f63a806b350 0x7f63a806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:52:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.702+0000 7f63c971f700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f63a806b350 0x7f63a806d810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000c00 tx=0x7f63c0008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:52:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.706+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63c4140500 con 0x7f63c415af10 2024-01-16T13:52:59.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.900+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c4140500 con 0x7f63c415af10 2024-01-16T13:52:59.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.901+0000 7f63b9ffb700 1 -- 172.21.15.113:0/2340139568 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16618 (secure 0 0 0) 0x7f63c4140500 con 0x7f63c415af10 2024-01-16T13:52:59.883 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:52:59.883 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":32,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:58.941518+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:56.889692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:57.908171+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f63a806b350 msgr2=0x7f63a806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f63a806b350 0x7f63a806d810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000c00 tx=0x7f63c0008030 comp rx=0 tx=0).stop 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 msgr2=0x7f63c41f0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c41f0b60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c820 tx=0x7f63b400cb30 comp rx=0 tx=0).stop 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 shutdown_connections 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f63a806b350 0x7f63a806d810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f63c415f460 0x7f63c41f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63c40ce140 0x7f63c4074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 --2- 172.21.15.113:0/2340139568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63c415af10 0x7f63c41f0b60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:52:59.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 >> 172.21.15.113:0/2340139568 conn(0x7f63c40cd570 msgr2=0x7f63c406dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:52:59.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 shutdown_connections 2024-01-16T13:52:59.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:52:59.903+0000 7f63cb983700 1 -- 172.21.15.113:0/2340139568 wait complete. 2024-01-16T13:52:59.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:53:00.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.035+0000 7f51d342c700 1 Processor -- start 2024-01-16T13:53:00.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.036+0000 7f51d342c700 1 -- start start 2024-01-16T13:53:00.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.036+0000 7f51d342c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f51cc0ce4d0 0x7f51cc0d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.036+0000 7f51d342c700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc0d0150 con 0x7f51cc0ce910 2024-01-16T13:53:00.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.036+0000 7f51d342c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc0cee50 con 0x7f51cc15e080 2024-01-16T13:53:00.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.036+0000 7f51d342c700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51cc15e460 con 0x7f51cc0ce4d0 2024-01-16T13:53:00.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d19c9700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f51cc0ce910 0x7f51cc0d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56010/0 (socket says 172.21.15.113:56010) 2024-01-16T13:53:00.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d19c9700 1 -- 172.21.15.113:0/927739147 learned_addr learned my addr 172.21.15.113:0/927739147 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:00.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d09c7700 1 --2- 172.21.15.113:0/927739147 >> v2:172.21.15.124:3300/0 conn(0x7f51cc0ce4d0 0x7f51cc0d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607202794 0 0) 0x7f51cc0d0150 con 0x7f51cc0ce910 2024-01-16T13:53:00.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b80035b0 con 0x7f51cc0ce910 2024-01-16T13:53:00.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 691998203 0 0) 0x7f51b80035b0 con 0x7f51cc0ce910 2024-01-16T13:53:00.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 >> v2:172.21.15.124:3300/0 conn(0x7f51cc0ce4d0 msgr2=0x7f51cc0d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 --2- 172.21.15.113:0/927739147 >> v2:172.21.15.124:3300/0 conn(0x7f51cc0ce4d0 0x7f51cc0d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 >> v1:172.21.15.124:6789/0 conn(0x7f51cc15e080 legacy=0x7f51cc0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc0cf130 con 0x7f51cc0ce910 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f51c40019f0 con 0x7f51cc0ce910 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 >> v1:172.21.15.113:6789/0 conn(0x7f51cc0ce910 legacy=0x7f51cc0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51b8003a00 0x7f51b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8004850 0x7f51b8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8005210 0x7f51b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc15e460 con 0x7f51b8003a00 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c0b0 con 0x7f51b8004850 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c270 con 0x7f51b8005210 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d11c8700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51b8003a00 0x7f51b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d09c7700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8004850 0x7f51b8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d19c9700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8005210 0x7f51b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d11c8700 1 -- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8005210 msgr2=0x7f51b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d11c8700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8005210 0x7f51b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.037+0000 7f51d11c8700 1 -- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8004850 msgr2=0x7f51b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d11c8700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8004850 0x7f51b8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d11c8700 1 -- 172.21.15.113:0/927739147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b80035b0 con 0x7f51b8003a00 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d19c9700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8005210 0x7f51b800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d19c9700 1 -- 172.21.15.113:0/927739147 reap_dead start 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d11c8700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51b8003a00 0x7f51b8003e20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0017c0 tx=0x7f51b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b11661a3a82ad6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc0052d0 con 0x7f51b8003a00 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc005430 con 0x7f51b8003a00 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51c3fff700 1 -- 172.21.15.113:0/927739147 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc0038c0 con 0x7f51b8003a00 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 -- 172.21.15.113:0/927739147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51b8003a00 msgr2=0x7f51b8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51b8003a00 0x7f51b8003e20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0017c0 tx=0x7f51b8005880 comp rx=0 tx=0).stop 2024-01-16T13:53:00.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 -- 172.21.15.113:0/927739147 shutdown_connections 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 --2- 172.21.15.113:0/927739147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51b8003a00 0x7f51b8003e20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 -- 172.21.15.113:0/927739147 >> 172.21.15.113:0/927739147 conn(0x7f51cc073130 msgr2=0x7f51cc09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 -- 172.21.15.113:0/927739147 shutdown_connections 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.038+0000 7f51d342c700 1 -- 172.21.15.113:0/927739147 wait complete. 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 Processor -- start 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 -- start start 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8003a00 0x7f51cc14d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 0x7f51cc0c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc15a580 0x7f51cc0b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0c8f50 con 0x7f51cc15a030 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0c84d0 con 0x7f51cc15a580 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d342c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0c7f30 con 0x7f51b8003a00 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d09c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 0x7f51cc0c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d09c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 0x7f51cc0c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48958/0 (socket says 172.21.15.113:48958) 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.040+0000 7f51d09c7700 1 -- 172.21.15.113:0/1702187965 learned_addr learned my addr 172.21.15.113:0/1702187965 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d11c8700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8003a00 0x7f51cc14d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d19c9700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc15a580 0x7f51cc0b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d09c7700 1 -- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8003a00 msgr2=0x7f51cc14d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d09c7700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8003a00 0x7f51cc14d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d09c7700 1 -- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc15a580 msgr2=0x7f51cc0b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d09c7700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc15a580 0x7f51cc0b5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d09c7700 1 -- 172.21.15.113:0/1702187965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc0c6630 con 0x7f51cc15a030 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d09c7700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 0x7f51cc0c6ac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f51c8011aa0 tx=0x7f51c8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800f8e0 con 0x7f51cc15a030 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c800fa40 con 0x7f51cc15a030 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc14d5e0 con 0x7f51cc15a030 2024-01-16T13:53:00.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800fdf0 con 0x7f51cc15a030 2024-01-16T13:53:00.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.041+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc0c7c50 con 0x7f51cc15a030 2024-01-16T13:53:00.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.042+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc0c67c0 con 0x7f51cc15a030 2024-01-16T13:53:00.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.043+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f51c8014440 con 0x7f51cc15a030 2024-01-16T13:53:00.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.043+0000 7f51c1ffb700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f51b006b350 0x7f51b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.043+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51c808a0f0 con 0x7f51cc15a030 2024-01-16T13:53:00.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.043+0000 7f51d11c8700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f51b006b350 0x7f51b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.043+0000 7f51d11c8700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f51b006b350 0x7f51b006d810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f51bc000c00 tx=0x7f51bc009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:00.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.047+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51c8054ff0 con 0x7f51cc15a030 2024-01-16T13:53:00.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.239+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51cc067f60 con 0x7f51cc15a030 2024-01-16T13:53:00.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.240+0000 7f51c1ffb700 1 -- 172.21.15.113:0/1702187965 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16618 (secure 0 0 0) 0x7f51c8058610 con 0x7f51cc15a030 2024-01-16T13:53:00.222 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:00.222 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":32,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:52:58.941518+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:56.889692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:57.908171+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:53:00.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f51b006b350 msgr2=0x7f51b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f51b006b350 0x7f51b006d810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f51bc000c00 tx=0x7f51bc009030 comp rx=0 tx=0).stop 2024-01-16T13:53:00.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 msgr2=0x7f51cc0c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 0x7f51cc0c6ac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f51c8011aa0 tx=0x7f51c8011db0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 shutdown_connections 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f51b006b350 0x7f51b006d810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc15a580 0x7f51cc0b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51b8003a00 0x7f51cc14d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 --2- 172.21.15.113:0/1702187965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51cc15a030 0x7f51cc0c6ac0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 >> 172.21.15.113:0/1702187965 conn(0x7f51cc073130 msgr2=0x7f51cc06feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 shutdown_connections 2024-01-16T13:53:00.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.242+0000 7f51d342c700 1 -- 172.21.15.113:0/1702187965 wait complete. 2024-01-16T13:53:00.236 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T13:53:00.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.378+0000 7f63998e2700 1 Processor -- start 2024-01-16T13:53:00.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.378+0000 7f63998e2700 1 -- start start 2024-01-16T13:53:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63998e2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63940cf330 0x7f63940d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63998e2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f63940b98e0 0x7f63940d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63998e2700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6394162af0 con 0x7f6394161510 2024-01-16T13:53:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63998e2700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63940d13e0 con 0x7f63940b98e0 2024-01-16T13:53:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63998e2700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6394161f70 con 0x7f63940cf330 2024-01-16T13:53:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f6392ffd700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6394161510 0x7f63940cec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56016/0 (socket says 172.21.15.113:56016) 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f6392ffd700 1 -- 172.21.15.113:0/2514994165 learned_addr learned my addr 172.21.15.113:0/2514994165 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63937fe700 1 --2- 172.21.15.113:0/2514994165 >> v2:172.21.15.113:3300/0 conn(0x7f63940b98e0 0x7f63940d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63927fc700 1 --2- 172.21.15.113:0/2514994165 >> v2:172.21.15.124:3300/0 conn(0x7f63940cf330 0x7f63940d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63937fe700 1 -- 172.21.15.113:0/2514994165 >> v2:172.21.15.124:3300/0 conn(0x7f63940cf330 msgr2=0x7f63940d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63937fe700 1 --2- 172.21.15.113:0/2514994165 >> v2:172.21.15.124:3300/0 conn(0x7f63940cf330 0x7f63940d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63937fe700 1 -- 172.21.15.113:0/2514994165 >> v1:172.21.15.113:6789/0 conn(0x7f6394161510 legacy=0x7f63940cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63937fe700 1 -- 172.21.15.113:0/2514994165 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394162250 con 0x7f63940b98e0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f6391ffb700 1 -- 172.21.15.113:0/2514994165 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642527023 0 0) 0x7f6394162af0 con 0x7f6394161510 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.379+0000 7f63937fe700 1 --2- 172.21.15.113:0/2514994165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940b98e0 0x7f63940d2c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f63940ce2c0 tx=0x7f638c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e09907f73bab61 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f6391ffb700 1 -- 172.21.15.113:0/2514994165 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c00e070 con 0x7f63940b98e0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f6391ffb700 1 -- 172.21.15.113:0/2514994165 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c00bbd0 con 0x7f63940b98e0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f6391ffb700 1 -- 172.21.15.113:0/2514994165 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c012430 con 0x7f63940b98e0 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 -- 172.21.15.113:0/2514994165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940b98e0 msgr2=0x7f63940d2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 --2- 172.21.15.113:0/2514994165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940b98e0 0x7f63940d2c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f63940ce2c0 tx=0x7f638c00b330 comp rx=0 tx=0).stop 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 -- 172.21.15.113:0/2514994165 shutdown_connections 2024-01-16T13:53:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 --2- 172.21.15.113:0/2514994165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940b98e0 0x7f63940d2c40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 --2- 172.21.15.113:0/2514994165 >> v2:172.21.15.124:3300/0 conn(0x7f63940cf330 0x7f63940d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 -- 172.21.15.113:0/2514994165 >> 172.21.15.113:0/2514994165 conn(0x7f639406b410 msgr2=0x7f639406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:00.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 -- 172.21.15.113:0/2514994165 shutdown_connections 2024-01-16T13:53:00.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.380+0000 7f63998e2700 1 -- 172.21.15.113:0/2514994165 wait complete. 2024-01-16T13:53:00.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.381+0000 7f63998e2700 1 Processor -- start 2024-01-16T13:53:00.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 -- start start 2024-01-16T13:53:00.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63940b98e0 0x7f63941482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 0x7f639415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6394161510 0x7f63940b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63941487e0 con 0x7f63940cf330 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63940b39b0 con 0x7f63940b98e0 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63998e2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63940b3c90 con 0x7f6394161510 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.382+0000 7f63927fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 0x7f639415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 0x7f639415c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48970/0 (socket says 172.21.15.113:48970) 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 -- 172.21.15.113:0/2258712595 learned_addr learned my addr 172.21.15.113:0/2258712595 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63937fe700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6394161510 0x7f63940b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f6392ffd700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63940b98e0 0x7f63941482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 -- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6394161510 msgr2=0x7f63940b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6394161510 0x7f63940b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 -- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63940b98e0 msgr2=0x7f63941482a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63940b98e0 0x7f63941482a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 -- 172.21.15.113:0/2258712595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c009a70 con 0x7f63940cf330 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f6392ffd700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63940b98e0 0x7f63941482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:00.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63927fc700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 0x7f639415c680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f638400e810 tx=0x7f638400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384013610 con 0x7f63940cf330 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6384013770 con 0x7f63940cf330 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63940b3f70 con 0x7f63940cf330 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384013b20 con 0x7f63940cf330 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.383+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63940b4530 con 0x7f63940cf330 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.384+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6394068450 con 0x7f63940cf330 2024-01-16T13:53:00.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.385+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6384013c80 con 0x7f63940cf330 2024-01-16T13:53:00.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.385+0000 7f637bfff700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f638006b380 0x7f638006d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:00.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.385+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6384089790 con 0x7f63940cf330 2024-01-16T13:53:00.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.385+0000 7f6392ffd700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f638006b380 0x7f638006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:00.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.385+0000 7f6392ffd700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f638006b380 0x7f638006d840 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6388000f60 tx=0x7f6388007d30 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:00.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.389+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6384054610 con 0x7f63940cf330 2024-01-16T13:53:00.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:00.628+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f63940b4750 con 0x7f63940cf330 2024-01-16T13:53:01.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.565+0000 7f637bfff700 1 -- 172.21.15.113:0/2258712595 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6384057c30 con 0x7f63940cf330 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f638006b380 msgr2=0x7f638006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f638006b380 0x7f638006d840 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6388000f60 tx=0x7f6388007d30 comp rx=0 tx=0).stop 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 msgr2=0x7f639415c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 0x7f639415c680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f638400e810 tx=0x7f638400eb20 comp rx=0 tx=0).stop 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 shutdown_connections 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f638006b380 0x7f638006d840 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6394161510 0x7f63940b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63940b98e0 0x7f63941482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 --2- 172.21.15.113:0/2258712595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940cf330 0x7f639415c680 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 >> 172.21.15.113:0/2258712595 conn(0x7f639406b410 msgr2=0x7f639415eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:01.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 shutdown_connections 2024-01-16T13:53:01.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.568+0000 7f63998e2700 1 -- 172.21.15.113:0/2258712595 wait complete. 2024-01-16T13:53:01.564 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T13:53:01.565 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T13:53:01.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.712+0000 7fe034347700 1 Processor -- start 2024-01-16T13:53:01.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.712+0000 7fe034347700 1 -- start start 2024-01-16T13:53:01.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe034347700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe02c0ceb60 0x7fe02c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe034347700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c06edd0 con 0x7fe02c06fe50 2024-01-16T13:53:01.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe034347700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c06f090 con 0x7fe02c0cdbc0 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe034347700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c06f370 con 0x7fe02c0ceb60 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe0318e2700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fe02c06fe50 0x7fe02c0ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56024/0 (socket says 172.21.15.113:56024) 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe0318e2700 1 -- 172.21.15.113:0/619742374 learned_addr learned my addr 172.21.15.113:0/619742374 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155686577 0 0) 0x7fe02c06edd0 con 0x7fe02c06fe50 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0180035b0 con 0x7fe02c06fe50 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.713+0000 7fe0328e4700 1 --2- 172.21.15.113:0/619742374 >> v2:172.21.15.124:3300/0 conn(0x7fe02c0ceb60 0x7fe02c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 100365196 0 0) 0x7fe0180035b0 con 0x7fe02c06fe50 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 >> v2:172.21.15.124:3300/0 conn(0x7fe02c0ceb60 msgr2=0x7fe02c15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 --2- 172.21.15.113:0/619742374 >> v2:172.21.15.124:3300/0 conn(0x7fe02c0ceb60 0x7fe02c15d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 >> v1:172.21.15.124:6789/0 conn(0x7fe02c0cdbc0 legacy=0x7fe02c0d0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c06f650 con 0x7fe02c06fe50 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fe028001670 con 0x7fe02c06fe50 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 >> v1:172.21.15.113:6789/0 conn(0x7fe02c06fe50 legacy=0x7fe02c0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe018003a00 0x7fe018003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe018004850 0x7fe018004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018005210 0x7fe01800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c06f370 con 0x7fe018003a00 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01800c0b0 con 0x7fe018004850 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01800c270 con 0x7fe018005210 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe018003a00 0x7fe018003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0318e2700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe018004850 0x7fe018004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0328e4700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018005210 0x7fe01800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 -- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018005210 msgr2=0x7fe01800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018005210 0x7fe01800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 -- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe018004850 msgr2=0x7fe018004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe018004850 0x7fe018004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 -- 172.21.15.113:0/619742374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180035b0 con 0x7fe018003a00 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0328e4700 1 -- 172.21.15.113:0/619742374 reap_dead start 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0320e3700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe018003a00 0x7fe018003e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0017c0 tx=0x7fe018005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba18553a3ecafe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0052d0 con 0x7fe018003a00 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c005450 con 0x7fe018003a00 2024-01-16T13:53:01.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.714+0000 7fe0310e1700 1 -- 172.21.15.113:0/619742374 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0038c0 con 0x7fe018003a00 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 -- 172.21.15.113:0/619742374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe018003a00 msgr2=0x7fe018003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe018003a00 0x7fe018003e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0017c0 tx=0x7fe018005880 comp rx=0 tx=0).stop 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 -- 172.21.15.113:0/619742374 shutdown_connections 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 --2- 172.21.15.113:0/619742374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe018003a00 0x7fe018003e20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 -- 172.21.15.113:0/619742374 >> 172.21.15.113:0/619742374 conn(0x7fe02c0b3780 msgr2=0x7fe02c0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 -- 172.21.15.113:0/619742374 shutdown_connections 2024-01-16T13:53:01.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.715+0000 7fe034347700 1 -- 172.21.15.113:0/619742374 wait complete. 2024-01-16T13:53:01.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.716+0000 7fe034347700 1 Processor -- start 2024-01-16T13:53:01.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.716+0000 7fe034347700 1 -- start start 2024-01-16T13:53:01.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe034347700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018003a00 0x7fe02c0b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe034347700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe02c0b3dd0 0x7fe02c0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe034347700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 0x7fe02c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe034347700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0b5b70 con 0x7fe02c0b4250 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe034347700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0b4870 con 0x7fe02c0b3dd0 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe034347700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0b4b50 con 0x7fe018003a00 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 0x7fe02c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 0x7fe02c156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48982/0 (socket says 172.21.15.113:48982) 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 -- 172.21.15.113:0/775919333 learned_addr learned my addr 172.21.15.113:0/775919333 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0320e3700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018003a00 0x7fe02c0b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0318e2700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe02c0b3dd0 0x7fe02c0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 -- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018003a00 msgr2=0x7fe02c0b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018003a00 0x7fe02c0b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 -- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe02c0b3dd0 msgr2=0x7fe02c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe02c0b3dd0 0x7fe02c0b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.717+0000 7fe0328e4700 1 -- 172.21.15.113:0/775919333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0b4e30 con 0x7fe02c0b4250 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.718+0000 7fe0328e4700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 0x7fe02c156f50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe02c072120 tx=0x7fe024011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:01.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.718+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02400e040 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.718+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400f040 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.718+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02400fd00 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.718+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c0b5110 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.718+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c0b69c0 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.719+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c0b5300 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.719+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe024022460 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.720+0000 7fe022ffd700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe01006b350 0x7fe01006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.720+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe02408a080 con 0x7fe02c0b4250 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.720+0000 7fe0320e3700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe01006b350 0x7fe01006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:01.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.720+0000 7fe0320e3700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe01006b350 0x7fe01006d810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe01c000c00 tx=0x7fe01c009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:01.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.724+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe024054ef0 con 0x7fe02c0b4250 2024-01-16T13:53:01.923 INFO:tasks.ceph.mds.d.smithi124.stderr:2024-01-16T13:53:01.941+0000 7fc104620700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T13:53:01.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:01.960+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe02c069020 con 0x7fe02c0b4250 2024-01-16T13:53:02.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.570+0000 7fe022ffd700 1 -- 172.21.15.113:0/775919333 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe024058510 con 0x7fe02c0b4250 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe01006b350 msgr2=0x7fe01006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe01006b350 0x7fe01006d810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe01c000c00 tx=0x7fe01c009030 comp rx=0 tx=0).stop 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 msgr2=0x7fe02c156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 0x7fe02c156f50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe02c072120 tx=0x7fe024011ed0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 shutdown_connections 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fe01006b350 0x7fe01006d810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe02c0b4250 0x7fe02c156f50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe018003a00 0x7fe02c0b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 --2- 172.21.15.113:0/775919333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe02c0b3dd0 0x7fe02c0b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 >> 172.21.15.113:0/775919333 conn(0x7fe02c0b3780 msgr2=0x7fe02c0b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.572+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 shutdown_connections 2024-01-16T13:53:02.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.573+0000 7fe034347700 1 -- 172.21.15.113:0/775919333 wait complete. 2024-01-16T13:53:02.567 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T13:53:02.568 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:53:02.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.707+0000 7fdf47826700 1 Processor -- start 2024-01-16T13:53:02.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.708+0000 7fdf47826700 1 -- start start 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.708+0000 7fdf47826700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdf401473a0 0x7fdf4015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.708+0000 7fdf47826700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4015eb00 con 0x7fdf40147780 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf47826700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4015ede0 con 0x7fdf4006dc20 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf47826700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4015f0c0 con 0x7fdf401473a0 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf455c2700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdf401473a0 0x7fdf4015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf44dc1700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fdf40147780 0x7fdf4006c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56026/0 (socket says 172.21.15.113:56026) 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf44dc1700 1 -- 172.21.15.113:0/277038589 learned_addr learned my addr 172.21.15.113:0/277038589 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf455c2700 1 -- 172.21.15.113:0/277038589 >> v1:172.21.15.124:6789/0 conn(0x7fdf4006dc20 legacy=0x7fdf40073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf455c2700 1 -- 172.21.15.113:0/277038589 >> v1:172.21.15.113:6789/0 conn(0x7fdf40147780 legacy=0x7fdf4006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf455c2700 1 -- 172.21.15.113:0/277038589 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4015f3a0 con 0x7fdf401473a0 2024-01-16T13:53:02.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf37fff700 1 -- 172.21.15.113:0/277038589 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460650038 0 0) 0x7fdf4015eb00 con 0x7fdf40147780 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf455c2700 1 --2- 172.21.15.113:0/277038589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf4015e4b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009840 tx=0x7fdf30009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433995d166508cf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf37fff700 1 -- 172.21.15.113:0/277038589 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30004230 con 0x7fdf401473a0 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf37fff700 1 -- 172.21.15.113:0/277038589 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf30004390 con 0x7fdf401473a0 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.709+0000 7fdf37fff700 1 -- 172.21.15.113:0/277038589 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf300036a0 con 0x7fdf401473a0 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 -- 172.21.15.113:0/277038589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 msgr2=0x7fdf4015e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 --2- 172.21.15.113:0/277038589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf4015e4b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009840 tx=0x7fdf30009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 -- 172.21.15.113:0/277038589 shutdown_connections 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 --2- 172.21.15.113:0/277038589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf4015e4b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 -- 172.21.15.113:0/277038589 >> 172.21.15.113:0/277038589 conn(0x7fdf400b3130 msgr2=0x7fdf400d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 -- 172.21.15.113:0/277038589 shutdown_connections 2024-01-16T13:53:02.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.710+0000 7fdf47826700 1 -- 172.21.15.113:0/277038589 wait complete. 2024-01-16T13:53:02.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.711+0000 7fdf47826700 1 Processor -- start 2024-01-16T13:53:02.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.711+0000 7fdf47826700 1 -- start start 2024-01-16T13:53:02.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4006dc20 0x7fdf4015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:02.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf400cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:02.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf40147780 0x7fdf400cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400ced00 con 0x7fdf401473a0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4015d9e0 con 0x7fdf4006dc20 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4015d440 con 0x7fdf40147780 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf400cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf400cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48994/0 (socket says 172.21.15.113:48994) 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 -- 172.21.15.113:0/1028501217 learned_addr learned my addr 172.21.15.113:0/1028501217 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf455c2700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4006dc20 0x7fdf4015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf45dc3700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf40147780 0x7fdf400cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 -- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf40147780 msgr2=0x7fdf400cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf40147780 0x7fdf400cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 -- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4006dc20 msgr2=0x7fdf4015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4006dc20 0x7fdf4015cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 -- 172.21.15.113:0/1028501217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf300094a0 con 0x7fdf401473a0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf44dc1700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf400cd570 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c520 tx=0x7fdf3c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c004c80 con 0x7fdf401473a0 2024-01-16T13:53:02.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c004de0 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf400ce0f0 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.712+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c0072d0 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.713+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf400ce6b0 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.714+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf400ceeb0 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.714+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdf3c007430 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.714+0000 7fdf35ffb700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdf2406b350 0x7fdf2406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.714+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf3c089910 con 0x7fdf401473a0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.714+0000 7fdf455c2700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdf2406b350 0x7fdf2406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:02.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.715+0000 7fdf455c2700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdf2406b350 0x7fdf2406d810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009b90 tx=0x7fdf30008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:02.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.719+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf400ceeb0 con 0x7fdf401473a0 2024-01-16T13:53:02.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.953+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf40067f60 con 0x7fdf401473a0 2024-01-16T13:53:02.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.954+0000 7fdf35ffb700 1 -- 172.21.15.113:0/1028501217 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4925 (secure 0 0 0) 0x7fdf3c054770 con 0x7fdf401473a0 2024-01-16T13:53:02.935 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:02.936 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4915,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/2577054536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2577054536},{"type":"v1","addr":"172.21.15.124:6834","nonce":2577054536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2172474833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":2172474833},{"type":"v1","addr":"172.21.15.113:6836","nonce":2172474833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5058,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/2409334271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2409334271},{"type":"v1","addr":"172.21.15.113:6837","nonce":2409334271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:58.941819+0000","modified":"2024-01-16T13:53:01.960866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.124:6835/1874427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":1874427660},{"type":"v1","addr":"172.21.15.124:6835","nonce":1874427660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdf2406b350 msgr2=0x7fdf2406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdf2406b350 0x7fdf2406d810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009b90 tx=0x7fdf30008030 comp rx=0 tx=0).stop 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 msgr2=0x7fdf400cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf400cd570 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c520 tx=0x7fdf3c00c8e0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 shutdown_connections 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdf2406b350 0x7fdf2406d810 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf40147780 0x7fdf400cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4006dc20 0x7fdf4015cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 --2- 172.21.15.113:0/1028501217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf401473a0 0x7fdf400cd570 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 >> 172.21.15.113:0/1028501217 conn(0x7fdf400b3130 msgr2=0x7fdf40073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:02.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 shutdown_connections 2024-01-16T13:53:02.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:02.956+0000 7fdf47826700 1 -- 172.21.15.113:0/1028501217 wait complete. 2024-01-16T13:53:02.939 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 16 2024-01-16T13:53:02.950 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:53:03.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.088+0000 7f036ebf1700 1 Processor -- start 2024-01-16T13:53:03.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.088+0000 7f036ebf1700 1 -- start start 2024-01-16T13:53:03.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f036ebf1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 0x7f03680b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f036ebf1700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680af950 con 0x7f03680adf00 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f036ebf1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680afc30 con 0x7f036806c710 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f036ebf1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0368144a80 con 0x7f03681446a0 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f0367fff700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f03680adf00 0x7f036806c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:56028/0 (socket says 172.21.15.113:56028) 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f0367fff700 1 -- 172.21.15.113:0/388524855 learned_addr learned my addr 172.21.15.113:0/388524855 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029174001 0 0) 0x7f03680af950 con 0x7f03680adf00 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f036c98d700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 0x7f03680b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.089+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03580035b0 con 0x7f03680adf00 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075776046 0 0) 0x7f03580035b0 con 0x7f03680adf00 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 msgr2=0x7f03680b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 0x7f03680b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 >> v1:172.21.15.124:6789/0 conn(0x7f036806c710 legacy=0x7f03680b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680aff10 con 0x7f03680adf00 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f0354001670 con 0x7f03680adf00 2024-01-16T13:53:03.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 >> v1:172.21.15.113:6789/0 conn(0x7f03680adf00 legacy=0x7f036806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0358003a30 0x7f0358003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f0358004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 0x7f0358005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0368144a80 con 0x7f0358004850 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035800bac0 con 0x7f0358003a30 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035800de90 con 0x7f03681446a0 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f0358004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f036c98d700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0358003a30 0x7f0358003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f036d18e700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 0x7f0358005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 -- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 msgr2=0x7f0358005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03681446a0 0x7f0358005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 -- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0358003a30 msgr2=0x7f0358003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0358003a30 0x7f0358003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 -- 172.21.15.113:0/388524855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03580035b0 con 0x7f0358004850 2024-01-16T13:53:03.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f0367fff700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f0358004cd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0354001eb0 tx=0x7f0358005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f6f0d70e9ff73a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035400cd50 con 0x7f0358004850 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0354005960 con 0x7f0358004850 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.090+0000 7f03677fe700 1 -- 172.21.15.113:0/388524855 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354005d10 con 0x7f0358004850 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036ebf1700 1 -- 172.21.15.113:0/388524855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 msgr2=0x7f0358004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036ebf1700 1 --2- 172.21.15.113:0/388524855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f0358004cd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0354001eb0 tx=0x7f0358005880 comp rx=0 tx=0).stop 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036d18e700 1 -- 172.21.15.113:0/388524855 reap_dead start 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036ebf1700 1 -- 172.21.15.113:0/388524855 shutdown_connections 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036ebf1700 1 -- 172.21.15.113:0/388524855 >> 172.21.15.113:0/388524855 conn(0x7f0368071b90 msgr2=0x7f0368071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036ebf1700 1 -- 172.21.15.113:0/388524855 shutdown_connections 2024-01-16T13:53:03.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.091+0000 7f036ebf1700 1 -- 172.21.15.113:0/388524855 wait complete. 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.092+0000 7f036ebf1700 1 Processor -- start 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.092+0000 7f036ebf1700 1 -- start start 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036ebf1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f036813f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036ebf1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f036813f820 0x7f03680cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036ebf1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03680cfac0 0x7f03680cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036ebf1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680d0470 con 0x7f0358004850 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036ebf1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036813f560 con 0x7f036813f820 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036ebf1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680ce000 con 0x7f03680cfac0 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036c98d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f036813f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036c98d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f036813f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49012/0 (socket says 172.21.15.113:49012) 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036c98d700 1 -- 172.21.15.113:0/441463162 learned_addr learned my addr 172.21.15.113:0/441463162 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f036d18e700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03680cfac0 0x7f03680cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.093+0000 7f0367fff700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f036813f820 0x7f03680cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036c98d700 1 -- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03680cfac0 msgr2=0x7f03680cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036c98d700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03680cfac0 0x7f03680cff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036c98d700 1 -- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f036813f820 msgr2=0x7f03680cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036c98d700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f036813f820 0x7f03680cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036c98d700 1 -- 172.21.15.113:0/441463162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680ce2e0 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f0367fff700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f036813f820 0x7f03680cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036c98d700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f036813f020 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f035c011aa0 tx=0x7f035c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c00f980 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c00fae0 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c00fe90 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03680ce5c0 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.094+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03680ceb80 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.095+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03680753f0 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.095+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f035c014440 con 0x7f0358004850 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.096+0000 7f03657fa700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f034c06b350 0x7f034c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.096+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f035c08a2d0 con 0x7f0358004850 2024-01-16T13:53:03.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.096+0000 7f0367fff700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f034c06b350 0x7f034c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.096+0000 7f0367fff700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f034c06b350 0x7f034c06d810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0354001eb0 tx=0x7f035400a000 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.100+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f035c055290 con 0x7f0358004850 2024-01-16T13:53:03.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.291+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03680ceda0 con 0x7f0358004850 2024-01-16T13:53:03.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.292+0000 7f03657fa700 1 -- 172.21.15.113:0/441463162 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f035c0588b0 con 0x7f0358004850 2024-01-16T13:53:03.274 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:03.274 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":33,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:53:00.551594+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T13:52:56.889692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T13:52:57.908171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:53:03.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f034c06b350 msgr2=0x7f034c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f034c06b350 0x7f034c06d810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0354001eb0 tx=0x7f035400a000 comp rx=0 tx=0).stop 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 msgr2=0x7f036813f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f036813f020 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f035c011aa0 tx=0x7f035c011e60 comp rx=0 tx=0).stop 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 shutdown_connections 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f034c06b350 0x7f034c06d810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03680cfac0 0x7f03680cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0358004850 0x7f036813f020 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 --2- 172.21.15.113:0/441463162 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f036813f820 0x7f03680cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 >> 172.21.15.113:0/441463162 conn(0x7f0368071b90 msgr2=0x7f0368071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 shutdown_connections 2024-01-16T13:53:03.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.294+0000 7f036ebf1700 1 -- 172.21.15.113:0/441463162 wait complete. 2024-01-16T13:53:03.288 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T13:53:03.289 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:53:03.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.428+0000 7fc5bf775700 1 Processor -- start 2024-01-16T13:53:03.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.428+0000 7fc5bf775700 1 -- start start 2024-01-16T13:53:03.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bf775700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5b80ce4d0 0x7fc5b80d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bf775700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80d0150 con 0x7fc5b80ce910 2024-01-16T13:53:03.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bf775700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80cee50 con 0x7fc5b815e080 2024-01-16T13:53:03.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bf775700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b815e460 con 0x7fc5b80ce4d0 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bcd10700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5b80ce4d0 0x7fc5b80d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bdd12700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc5b80ce910 0x7fc5b80d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:45244/0 (socket says 172.21.15.113:45244) 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bcd10700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5b80ce4d0 0x7fc5b80d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49934/0 (socket says 172.21.15.113:49934) 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bdd12700 1 -- 172.21.15.113:0/29523774 learned_addr learned my addr 172.21.15.113:0/29523774 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bcd10700 1 -- 172.21.15.113:0/29523774 >> v1:172.21.15.124:6789/0 conn(0x7fc5b815e080 legacy=0x7fc5b80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bcd10700 1 -- 172.21.15.113:0/29523774 >> v1:172.21.15.113:6789/0 conn(0x7fc5b80ce910 legacy=0x7fc5b80d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bcd10700 1 -- 172.21.15.113:0/29523774 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80cf130 con 0x7fc5b80ce4d0 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5affff700 1 -- 172.21.15.113:0/29523774 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014637180 0 0) 0x7fc5b80d0150 con 0x7fc5b80ce910 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5bcd10700 1 --2- 172.21.15.113:0/29523774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80d1be0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009840 tx=0x7fc5a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b2a9bf10843662 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.429+0000 7fc5affff700 1 -- 172.21.15.113:0/29523774 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8004230 con 0x7fc5b80ce4d0 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5affff700 1 -- 172.21.15.113:0/29523774 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a8004390 con 0x7fc5b80ce4d0 2024-01-16T13:53:03.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5affff700 1 -- 172.21.15.113:0/29523774 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80036a0 con 0x7fc5b80ce4d0 2024-01-16T13:53:03.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 -- 172.21.15.113:0/29523774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80ce4d0 msgr2=0x7fc5b80d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 --2- 172.21.15.113:0/29523774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80d1be0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009840 tx=0x7fc5a8009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:03.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 -- 172.21.15.113:0/29523774 shutdown_connections 2024-01-16T13:53:03.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 --2- 172.21.15.113:0/29523774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80d1be0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 -- 172.21.15.113:0/29523774 >> 172.21.15.113:0/29523774 conn(0x7fc5b8073130 msgr2=0x7fc5b809aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:03.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 -- 172.21.15.113:0/29523774 shutdown_connections 2024-01-16T13:53:03.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.430+0000 7fc5bf775700 1 -- 172.21.15.113:0/29523774 wait complete. 2024-01-16T13:53:03.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.431+0000 7fc5bf775700 1 Processor -- start 2024-01-16T13:53:03.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.431+0000 7fc5bf775700 1 -- start start 2024-01-16T13:53:03.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bf775700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80b6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bf775700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5b80ce910 0x7fc5b81593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bf775700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 0x7fc5b8159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bf775700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c9a10 con 0x7fc5b815e080 2024-01-16T13:53:03.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bf775700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c9cf0 con 0x7fc5b80ce4d0 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bf775700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c9fd0 con 0x7fc5b80ce910 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 0x7fc5b8159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 0x7fc5b8159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49942/0 (socket says 172.21.15.113:49942) 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 -- 172.21.15.113:0/2072827336 learned_addr learned my addr 172.21.15.113:0/2072827336 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bcd10700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5b80ce910 0x7fc5b81593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bd511700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80b6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 -- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5b80ce910 msgr2=0x7fc5b81593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5b80ce910 0x7fc5b81593d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 -- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5b80ce4d0 msgr2=0x7fc5b80b6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80b6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.432+0000 7fc5bdd12700 1 -- 172.21.15.113:0/2072827336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80094a0 con 0x7fc5b815e080 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.433+0000 7fc5bdd12700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 0x7fc5b8159910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000cca0 tx=0x7fc5b000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.433+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000cea0 con 0x7fc5b815e080 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.433+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000ade0 con 0x7fc5b815e080 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.433+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80ca2b0 con 0x7fc5b815e080 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.433+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0007310 con 0x7fc5b815e080 2024-01-16T13:53:03.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.433+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b80caab0 con 0x7fc5b815e080 2024-01-16T13:53:03.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.434+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8067f60 con 0x7fc5b815e080 2024-01-16T13:53:03.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.434+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc5b0007470 con 0x7fc5b815e080 2024-01-16T13:53:03.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.434+0000 7fc5adffb700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc59c06b350 0x7fc59c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.435+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5b0089870 con 0x7fc5b815e080 2024-01-16T13:53:03.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.435+0000 7fc5bd511700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc59c06b350 0x7fc59c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.435+0000 7fc5bd511700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc59c06b350 0x7fc59c06d810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009870 tx=0x7fc5b4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.439+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5b00546e0 con 0x7fc5b815e080 2024-01-16T13:53:03.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.624+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5b80cc9b0 con 0x7fc59c06b350 2024-01-16T13:53:03.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.631+0000 7fc5adffb700 1 -- 172.21.15.113:0/2072827336 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+261242 (secure 0 0 0) 0x7fc5b40123e0 con 0x7fc59c06b350 2024-01-16T13:53:03.616 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc59c06b350 msgr2=0x7fc59c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc59c06b350 0x7fc59c06d810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009870 tx=0x7fc5b4008040 comp rx=0 tx=0).stop 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 msgr2=0x7fc5b8159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 0x7fc5b8159910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000cca0 tx=0x7fc5b000a340 comp rx=0 tx=0).stop 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 shutdown_connections 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc59c06b350 0x7fc59c06d810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b815e080 0x7fc5b8159910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5b80ce4d0 0x7fc5b80b6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 --2- 172.21.15.113:0/2072827336 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5b80ce910 0x7fc5b81593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.634+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 >> 172.21.15.113:0/2072827336 conn(0x7fc5b8073130 msgr2=0x7fc5b806fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:03.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.635+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 shutdown_connections 2024-01-16T13:53:03.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.635+0000 7fc5bf775700 1 -- 172.21.15.113:0/2072827336 wait complete. 2024-01-16T13:53:03.619 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:53:03.628 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-01-16T13:53:02.413046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219928,"kb_used_data":3688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754792,"statfs":{"total":773094113280,"available":772868907008,"internally_reserved":0,"allocated":3776512,"data_stored":2132168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.878047"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540461+0000","last_change":"2024-01-16T13:52:57.540461+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540461+0000","last_undegraded":"2024-01-16T13:52:57.540461+0000","last_fullsized":"2024-01-16T13:52:57.540461+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541729+0000","last_change":"2024-01-16T13:52:57.541729+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541729+0000","last_undegraded":"2024-01-16T13:52:57.541729+0000","last_fullsized":"2024-01-16T13:52:57.541729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541489+0000","last_change":"2024-01-16T13:52:57.541489+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541489+0000","last_undegraded":"2024-01-16T13:52:57.541489+0000","last_fullsized":"2024-01-16T13:52:57.541489+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538106+0000","last_change":"2024-01-16T13:52:57.538106+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538106+0000","last_undegraded":"2024-01-16T13:52:57.538106+0000","last_fullsized":"2024-01-16T13:52:57.538106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538125+0000","last_change":"2024-01-16T13:52:57.538125+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538125+0000","last_undegraded":"2024-01-16T13:52:57.538125+0000","last_fullsized":"2024-01-16T13:52:57.538125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542100+0000","last_change":"2024-01-16T13:52:57.542100+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542100+0000","last_undegraded":"2024-01-16T13:52:57.542100+0000","last_fullsized":"2024-01-16T13:52:57.542100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537122+0000","last_change":"2024-01-16T13:52:57.537122+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537122+0000","last_undegraded":"2024-01-16T13:52:57.537122+0000","last_fullsized":"2024-01-16T13:52:57.537122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541902+0000","last_change":"2024-01-16T13:52:57.541902+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541902+0000","last_undegraded":"2024-01-16T13:52:57.541902+0000","last_fullsized":"2024-01-16T13:52:57.541902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537171+0000","last_change":"2024-01-16T13:52:57.537171+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537171+0000","last_undegraded":"2024-01-16T13:52:57.537171+0000","last_fullsized":"2024-01-16T13:52:57.537171+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542291+0000","last_change":"2024-01-16T13:52:57.542291+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542291+0000","last_undegraded":"2024-01-16T13:52:57.542291+0000","last_fullsized":"2024-01-16T13:52:57.542291+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537449+0000","last_change":"2024-01-16T13:52:57.537449+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537449+0000","last_undegraded":"2024-01-16T13:52:57.537449+0000","last_fullsized":"2024-01-16T13:52:57.537449+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538504+0000","last_change":"2024-01-16T13:52:57.538504+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538504+0000","last_undegraded":"2024-01-16T13:52:57.538504+0000","last_fullsized":"2024-01-16T13:52:57.538504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541627+0000","last_change":"2024-01-16T13:52:57.541627+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541627+0000","last_undegraded":"2024-01-16T13:52:57.541627+0000","last_fullsized":"2024-01-16T13:52:57.541627+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542716+0000","last_change":"2024-01-16T13:52:57.542716+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542716+0000","last_undegraded":"2024-01-16T13:52:57.542716+0000","last_fullsized":"2024-01-16T13:52:57.542716+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538596+0000","last_change":"2024-01-16T13:52:57.538596+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538596+0000","last_undegraded":"2024-01-16T13:52:57.538596+0000","last_fullsized":"2024-01-16T13:52:57.538596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537813+0000","last_change":"2024-01-16T13:52:57.537813+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537813+0000","last_undegraded":"2024-01-16T13:52:57.537813+0000","last_fullsized":"2024-01-16T13:52:57.537813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537341+0000","last_change":"2024-01-16T13:52:57.537341+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537341+0000","last_undegraded":"2024-01-16T13:52:57.537341+0000","last_fullsized":"2024-01-16T13:52:57.537341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538104+0000","last_change":"2024-01-16T13:52:57.538104+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538104+0000","last_undegraded":"2024-01-16T13:52:57.538104+0000","last_fullsized":"2024-01-16T13:52:57.538104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540353+0000","last_change":"2024-01-16T13:52:57.540353+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540353+0000","last_undegraded":"2024-01-16T13:52:57.540353+0000","last_fullsized":"2024-01-16T13:52:57.540353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537698+0000","last_change":"2024-01-16T13:52:57.537698+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537698+0000","last_undegraded":"2024-01-16T13:52:57.537698+0000","last_fullsized":"2024-01-16T13:52:57.537698+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537078+0000","last_change":"2024-01-16T13:52:57.537078+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537078+0000","last_undegraded":"2024-01-16T13:52:57.537078+0000","last_fullsized":"2024-01-16T13:52:57.537078+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537842+0000","last_change":"2024-01-16T13:52:57.537842+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537842+0000","last_undegraded":"2024-01-16T13:52:57.537842+0000","last_fullsized":"2024-01-16T13:52:57.537842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537806+0000","last_change":"2024-01-16T13:52:57.537806+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537806+0000","last_undegraded":"2024-01-16T13:52:57.537806+0000","last_fullsized":"2024-01-16T13:52:57.537806+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537817+0000","last_change":"2024-01-16T13:52:57.537817+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537817+0000","last_undegraded":"2024-01-16T13:52:57.537817+0000","last_fullsized":"2024-01-16T13:52:57.537817+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537104+0000","last_change":"2024-01-16T13:52:57.537104+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537104+0000","last_undegraded":"2024-01-16T13:52:57.537104+0000","last_fullsized":"2024-01-16T13:52:57.537104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541601+0000","last_change":"2024-01-16T13:52:57.541601+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541601+0000","last_undegraded":"2024-01-16T13:52:57.541601+0000","last_fullsized":"2024-01-16T13:52:57.541601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540422+0000","last_change":"2024-01-16T13:52:57.540422+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540422+0000","last_undegraded":"2024-01-16T13:52:57.540422+0000","last_fullsized":"2024-01-16T13:52:57.540422+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-16T13:52:52.602144+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:52.602144+0000","last_peered":"2024-01-16T13:52:52.602144+0000","last_clean":"2024-01-16T13:52:52.602144+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:52.602144+0000","last_undegraded":"2024-01-16T13:52:52.602144+0000","last_fullsized":"2024-01-16T13:52:52.602144+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":917504,"data_stored":709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42199999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.628}]}]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.374,"5min":0.374,"15min":0.374},"last":0.34000000000000002}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"last":0.247},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.442}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.37},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26000000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32000000000000001}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.58699999999999997}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.48699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.254},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.32700000000000001}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.372}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38900000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41699999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.44700000000000001}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.55200000000000005}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.498},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:53:03.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:53:03.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.808+0000 7fd448709700 1 Processor -- start 2024-01-16T13:53:03.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.808+0000 7fd448709700 1 -- start start 2024-01-16T13:53:03.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd448709700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd44015e080 0x7fd4400cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd448709700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd4400d22f0 0x7fd4400d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd448709700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4400d0150 con 0x7fd4400ce4d0 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd448709700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd44015e460 con 0x7fd44015e080 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd448709700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4400cee50 con 0x7fd4400d22f0 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd44015e080 0x7fd4400cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd44015e080 0x7fd4400cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49956/0 (socket says 172.21.15.113:49956) 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 -- 172.21.15.113:0/2292692269 learned_addr learned my addr 172.21.15.113:0/2292692269 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 -- 172.21.15.113:0/2292692269 >> v2:172.21.15.124:3300/0 conn(0x7fd4400d22f0 msgr2=0x7fd4400d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 --2- 172.21.15.113:0/2292692269 >> v2:172.21.15.124:3300/0 conn(0x7fd4400d22f0 0x7fd4400d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 -- 172.21.15.113:0/2292692269 >> v1:172.21.15.124:6789/0 conn(0x7fd4400ce4d0 legacy=0x7fd4400d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.809+0000 7fd4464a5700 1 -- 172.21.15.113:0/2292692269 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400cf130 con 0x7fd44015e080 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd4464a5700 1 --2- 172.21.15.113:0/2292692269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd44015e080 0x7fd4400cc2a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd4400c8cb0 tx=0x7fd430009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc243d7cedf38b4f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd4454a3700 1 -- 172.21.15.113:0/2292692269 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430004230 con 0x7fd44015e080 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd4454a3700 1 -- 172.21.15.113:0/2292692269 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd430004390 con 0x7fd44015e080 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd4454a3700 1 -- 172.21.15.113:0/2292692269 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4300036a0 con 0x7fd44015e080 2024-01-16T13:53:03.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 -- 172.21.15.113:0/2292692269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd44015e080 msgr2=0x7fd4400cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 --2- 172.21.15.113:0/2292692269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd44015e080 0x7fd4400cc2a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd4400c8cb0 tx=0x7fd430009bd0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 -- 172.21.15.113:0/2292692269 shutdown_connections 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 --2- 172.21.15.113:0/2292692269 >> v2:172.21.15.124:3300/0 conn(0x7fd4400d22f0 0x7fd4400d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 --2- 172.21.15.113:0/2292692269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd44015e080 0x7fd4400cc2a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 -- 172.21.15.113:0/2292692269 >> 172.21.15.113:0/2292692269 conn(0x7fd440073130 msgr2=0x7fd44009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.810+0000 7fd448709700 1 -- 172.21.15.113:0/2292692269 shutdown_connections 2024-01-16T13:53:03.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.811+0000 7fd448709700 1 -- 172.21.15.113:0/2292692269 wait complete. 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 Processor -- start 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 -- start start 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 0x7fd4400b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd4400d22f0 0x7fd4400c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd44015e080 0x7fd4400cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4400b60f0 con 0x7fd4400ce4d0 2024-01-16T13:53:03.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4400b5470 con 0x7fd4400d22f0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd448709700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4400b5750 con 0x7fd44015e080 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.812+0000 7fd4464a5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 0x7fd4400b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 0x7fd4400b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49972/0 (socket says 172.21.15.113:49972) 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 -- 172.21.15.113:0/2363512229 learned_addr learned my addr 172.21.15.113:0/2363512229 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd445ca4700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd4400d22f0 0x7fd4400c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 -- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd44015e080 msgr2=0x7fd4400cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd44015e080 0x7fd4400cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 -- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd4400d22f0 msgr2=0x7fd4400c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd4400d22f0 0x7fd4400c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 -- 172.21.15.113:0/2363512229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd430009550 con 0x7fd4400ce4d0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd4464a5700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 0x7fd4400b4e90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd4400cf7d0 tx=0x7fd430004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430003dc0 con 0x7fd4400ce4d0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43001f070 con 0x7fd4400ce4d0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4400b5a30 con 0x7fd4400ce4d0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430011870 con 0x7fd4400ce4d0 2024-01-16T13:53:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.813+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4400c94d0 con 0x7fd4400ce4d0 2024-01-16T13:53:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.814+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4400c9660 con 0x7fd4400ce4d0 2024-01-16T13:53:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.814+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd430021030 con 0x7fd4400ce4d0 2024-01-16T13:53:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.815+0000 7fd436ffd700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd42406b350 0x7fd42406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.815+0000 7fd445ca4700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd42406b350 0x7fd42406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.815+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd43008b370 con 0x7fd4400ce4d0 2024-01-16T13:53:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.815+0000 7fd445ca4700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd42406b350 0x7fd42406d810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd4401f7360 tx=0x7fd43c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:03.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:03.819+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd430056390 con 0x7fd4400ce4d0 2024-01-16T13:53:03.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.006+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4400c8a70 con 0x7fd42406b350 2024-01-16T13:53:03.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.012+0000 7fd436ffd700 1 -- 172.21.15.113:0/2363512229 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+261242 (secure 0 0 0) 0x7fd4400c8a70 con 0x7fd42406b350 2024-01-16T13:53:03.997 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:03.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd42406b350 msgr2=0x7fd42406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd42406b350 0x7fd42406d810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd4401f7360 tx=0x7fd43c008040 comp rx=0 tx=0).stop 2024-01-16T13:53:03.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 msgr2=0x7fd4400b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:03.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 0x7fd4400b4e90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd4400cf7d0 tx=0x7fd430004750 comp rx=0 tx=0).stop 2024-01-16T13:53:03.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 shutdown_connections 2024-01-16T13:53:03.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd42406b350 0x7fd42406d810 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd44015e080 0x7fd4400cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4400ce4d0 0x7fd4400b4e90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 --2- 172.21.15.113:0/2363512229 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd4400d22f0 0x7fd4400c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 >> 172.21.15.113:0/2363512229 conn(0x7fd440073130 msgr2=0x7fd44009ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 shutdown_connections 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.016+0000 7fd448709700 1 -- 172.21.15.113:0/2363512229 wait complete. 2024-01-16T13:53:04.000 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:53:04.009 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-01-16T13:53:02.413046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219928,"kb_used_data":3688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754792,"statfs":{"total":773094113280,"available":772868907008,"internally_reserved":0,"allocated":3776512,"data_stored":2132168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.878047"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540461+0000","last_change":"2024-01-16T13:52:57.540461+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540461+0000","last_undegraded":"2024-01-16T13:52:57.540461+0000","last_fullsized":"2024-01-16T13:52:57.540461+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541729+0000","last_change":"2024-01-16T13:52:57.541729+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541729+0000","last_undegraded":"2024-01-16T13:52:57.541729+0000","last_fullsized":"2024-01-16T13:52:57.541729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541489+0000","last_change":"2024-01-16T13:52:57.541489+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541489+0000","last_undegraded":"2024-01-16T13:52:57.541489+0000","last_fullsized":"2024-01-16T13:52:57.541489+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538106+0000","last_change":"2024-01-16T13:52:57.538106+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538106+0000","last_undegraded":"2024-01-16T13:52:57.538106+0000","last_fullsized":"2024-01-16T13:52:57.538106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538125+0000","last_change":"2024-01-16T13:52:57.538125+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538125+0000","last_undegraded":"2024-01-16T13:52:57.538125+0000","last_fullsized":"2024-01-16T13:52:57.538125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542100+0000","last_change":"2024-01-16T13:52:57.542100+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542100+0000","last_undegraded":"2024-01-16T13:52:57.542100+0000","last_fullsized":"2024-01-16T13:52:57.542100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537122+0000","last_change":"2024-01-16T13:52:57.537122+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537122+0000","last_undegraded":"2024-01-16T13:52:57.537122+0000","last_fullsized":"2024-01-16T13:52:57.537122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541902+0000","last_change":"2024-01-16T13:52:57.541902+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541902+0000","last_undegraded":"2024-01-16T13:52:57.541902+0000","last_fullsized":"2024-01-16T13:52:57.541902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537171+0000","last_change":"2024-01-16T13:52:57.537171+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537171+0000","last_undegraded":"2024-01-16T13:52:57.537171+0000","last_fullsized":"2024-01-16T13:52:57.537171+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542291+0000","last_change":"2024-01-16T13:52:57.542291+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542291+0000","last_undegraded":"2024-01-16T13:52:57.542291+0000","last_fullsized":"2024-01-16T13:52:57.542291+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537449+0000","last_change":"2024-01-16T13:52:57.537449+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537449+0000","last_undegraded":"2024-01-16T13:52:57.537449+0000","last_fullsized":"2024-01-16T13:52:57.537449+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538504+0000","last_change":"2024-01-16T13:52:57.538504+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538504+0000","last_undegraded":"2024-01-16T13:52:57.538504+0000","last_fullsized":"2024-01-16T13:52:57.538504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541627+0000","last_change":"2024-01-16T13:52:57.541627+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541627+0000","last_undegraded":"2024-01-16T13:52:57.541627+0000","last_fullsized":"2024-01-16T13:52:57.541627+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542716+0000","last_change":"2024-01-16T13:52:57.542716+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542716+0000","last_undegraded":"2024-01-16T13:52:57.542716+0000","last_fullsized":"2024-01-16T13:52:57.542716+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538596+0000","last_change":"2024-01-16T13:52:57.538596+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538596+0000","last_undegraded":"2024-01-16T13:52:57.538596+0000","last_fullsized":"2024-01-16T13:52:57.538596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537813+0000","last_change":"2024-01-16T13:52:57.537813+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537813+0000","last_undegraded":"2024-01-16T13:52:57.537813+0000","last_fullsized":"2024-01-16T13:52:57.537813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537341+0000","last_change":"2024-01-16T13:52:57.537341+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537341+0000","last_undegraded":"2024-01-16T13:52:57.537341+0000","last_fullsized":"2024-01-16T13:52:57.537341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538104+0000","last_change":"2024-01-16T13:52:57.538104+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538104+0000","last_undegraded":"2024-01-16T13:52:57.538104+0000","last_fullsized":"2024-01-16T13:52:57.538104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540353+0000","last_change":"2024-01-16T13:52:57.540353+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540353+0000","last_undegraded":"2024-01-16T13:52:57.540353+0000","last_fullsized":"2024-01-16T13:52:57.540353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537698+0000","last_change":"2024-01-16T13:52:57.537698+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537698+0000","last_undegraded":"2024-01-16T13:52:57.537698+0000","last_fullsized":"2024-01-16T13:52:57.537698+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537078+0000","last_change":"2024-01-16T13:52:57.537078+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537078+0000","last_undegraded":"2024-01-16T13:52:57.537078+0000","last_fullsized":"2024-01-16T13:52:57.537078+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537842+0000","last_change":"2024-01-16T13:52:57.537842+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537842+0000","last_undegraded":"2024-01-16T13:52:57.537842+0000","last_fullsized":"2024-01-16T13:52:57.537842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537806+0000","last_change":"2024-01-16T13:52:57.537806+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537806+0000","last_undegraded":"2024-01-16T13:52:57.537806+0000","last_fullsized":"2024-01-16T13:52:57.537806+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537817+0000","last_change":"2024-01-16T13:52:57.537817+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537817+0000","last_undegraded":"2024-01-16T13:52:57.537817+0000","last_fullsized":"2024-01-16T13:52:57.537817+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537104+0000","last_change":"2024-01-16T13:52:57.537104+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537104+0000","last_undegraded":"2024-01-16T13:52:57.537104+0000","last_fullsized":"2024-01-16T13:52:57.537104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541601+0000","last_change":"2024-01-16T13:52:57.541601+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541601+0000","last_undegraded":"2024-01-16T13:52:57.541601+0000","last_fullsized":"2024-01-16T13:52:57.541601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540422+0000","last_change":"2024-01-16T13:52:57.540422+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540422+0000","last_undegraded":"2024-01-16T13:52:57.540422+0000","last_fullsized":"2024-01-16T13:52:57.540422+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-16T13:52:52.602144+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:52.602144+0000","last_peered":"2024-01-16T13:52:52.602144+0000","last_clean":"2024-01-16T13:52:52.602144+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:52.602144+0000","last_undegraded":"2024-01-16T13:52:52.602144+0000","last_fullsized":"2024-01-16T13:52:52.602144+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":917504,"data_stored":709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42199999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.628}]}]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.374,"5min":0.374,"15min":0.374},"last":0.34000000000000002}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"last":0.247},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.442}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.37},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26000000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32000000000000001}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.58699999999999997}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.48699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.254},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.32700000000000001}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.372}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38900000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41699999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.44700000000000001}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.55200000000000005}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.498},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:53:04.013 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:53:04.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.188+0000 7fbc3d353700 1 Processor -- start 2024-01-16T13:53:04.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.188+0000 7fbc3d353700 1 -- start start 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc3d353700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc3806eb70 0x7fbc38070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc3d353700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbc3815b140 0x7fbc38071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc3d353700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380b3000 con 0x7fbc38158770 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc3d353700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380b32e0 con 0x7fbc3806eb70 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc3d353700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380b35c0 con 0x7fbc3815b140 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc367fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc3806eb70 0x7fbc38070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc367fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc3806eb70 0x7fbc38070bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49986/0 (socket says 172.21.15.113:49986) 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.189+0000 7fbc367fc700 1 -- 172.21.15.113:0/1975846892 learned_addr learned my addr 172.21.15.113:0/1975846892 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc377fe700 1 --2- 172.21.15.113:0/1975846892 >> v2:172.21.15.124:3300/0 conn(0x7fbc3815b140 0x7fbc38071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc367fc700 1 -- 172.21.15.113:0/1975846892 >> v2:172.21.15.124:3300/0 conn(0x7fbc3815b140 msgr2=0x7fbc38071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc367fc700 1 --2- 172.21.15.113:0/1975846892 >> v2:172.21.15.124:3300/0 conn(0x7fbc3815b140 0x7fbc38071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc367fc700 1 -- 172.21.15.113:0/1975846892 >> v1:172.21.15.124:6789/0 conn(0x7fbc38158770 legacy=0x7fbc3806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc367fc700 1 -- 172.21.15.113:0/1975846892 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380b38a0 con 0x7fbc3806eb70 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc377fe700 1 --2- 172.21.15.113:0/1975846892 >> v2:172.21.15.124:3300/0 conn(0x7fbc3815b140 0x7fbc38071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc367fc700 1 --2- 172.21.15.113:0/1975846892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3806eb70 0x7fbc38070bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806d2f0 tx=0x7fbc28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2380b7792142aa1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc35ffb700 1 -- 172.21.15.113:0/1975846892 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28004230 con 0x7fbc3806eb70 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc35ffb700 1 -- 172.21.15.113:0/1975846892 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28004390 con 0x7fbc3806eb70 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.190+0000 7fbc35ffb700 1 -- 172.21.15.113:0/1975846892 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc280036a0 con 0x7fbc3806eb70 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 -- 172.21.15.113:0/1975846892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3806eb70 msgr2=0x7fbc38070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 --2- 172.21.15.113:0/1975846892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3806eb70 0x7fbc38070bc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806d2f0 tx=0x7fbc28009bd0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 -- 172.21.15.113:0/1975846892 shutdown_connections 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 --2- 172.21.15.113:0/1975846892 >> v2:172.21.15.124:3300/0 conn(0x7fbc3815b140 0x7fbc38071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 --2- 172.21.15.113:0/1975846892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3806eb70 0x7fbc38070bc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 -- 172.21.15.113:0/1975846892 >> 172.21.15.113:0/1975846892 conn(0x7fbc380b1740 msgr2=0x7fbc380b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 -- 172.21.15.113:0/1975846892 shutdown_connections 2024-01-16T13:53:04.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.191+0000 7fbc3d353700 1 -- 172.21.15.113:0/1975846892 wait complete. 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.192+0000 7fbc3d353700 1 Processor -- start 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.192+0000 7fbc3d353700 1 -- start start 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc3d353700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc3806eb70 0x7fbc380ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc3d353700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc38158770 0x7fbc381551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc3d353700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 0x7fbc380b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc3d353700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc381556f0 con 0x7fbc3815b140 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc3d353700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc381f4070 con 0x7fbc38158770 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc3d353700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380adf00 con 0x7fbc3806eb70 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 0x7fbc380b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 0x7fbc380b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50002/0 (socket says 172.21.15.113:50002) 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 -- 172.21.15.113:0/367002924 learned_addr learned my addr 172.21.15.113:0/367002924 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc36ffd700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc3806eb70 0x7fbc380ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc367fc700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc38158770 0x7fbc381551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 -- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc3806eb70 msgr2=0x7fbc380ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc3806eb70 0x7fbc380ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 -- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc38158770 msgr2=0x7fbc381551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc38158770 0x7fbc381551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 -- 172.21.15.113:0/367002924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc28009550 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc367fc700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc38158770 0x7fbc381551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc36ffd700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc3806eb70 0x7fbc380ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.193+0000 7fbc377fe700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 0x7fbc380b0380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000cef0 tx=0x7fbc3000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.194+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3000ae50 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.194+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc30015440 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.194+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc380ae1e0 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.194+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc300157f0 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.194+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc380ae7a0 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.194+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3815ada0 con 0x7fbc3815b140 2024-01-16T13:53:04.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.195+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbc3000d430 con 0x7fbc3815b140 2024-01-16T13:53:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.195+0000 7fbc2f7fe700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1806b350 0x7fbc1806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.195+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc3008a200 con 0x7fbc3815b140 2024-01-16T13:53:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.195+0000 7fbc36ffd700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1806b350 0x7fbc1806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.196+0000 7fbc36ffd700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1806b350 0x7fbc1806d810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbc20003fc0 tx=0x7fbc20009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.200+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc30055070 con 0x7fbc3815b140 2024-01-16T13:53:04.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.388+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc380b41d0 con 0x7fbc1806b350 2024-01-16T13:53:04.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.395+0000 7fbc2f7fe700 1 -- 172.21.15.113:0/367002924 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+261242 (secure 0 0 0) 0x7fbc20003bb0 con 0x7fbc1806b350 2024-01-16T13:53:04.379 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:04.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.397+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1806b350 msgr2=0x7fbc1806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.397+0000 7fbc3d353700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1806b350 0x7fbc1806d810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbc20003fc0 tx=0x7fbc20009040 comp rx=0 tx=0).stop 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 msgr2=0x7fbc380b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 0x7fbc380b0380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbc3000cef0 tx=0x7fbc3000a340 comp rx=0 tx=0).stop 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 shutdown_connections 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1806b350 0x7fbc1806d810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc3815b140 0x7fbc380b0380 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc3806eb70 0x7fbc380ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 --2- 172.21.15.113:0/367002924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc38158770 0x7fbc381551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 >> 172.21.15.113:0/367002924 conn(0x7fbc380b1740 msgr2=0x7fbc38070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 shutdown_connections 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.398+0000 7fbc3d353700 1 -- 172.21.15.113:0/367002924 wait complete. 2024-01-16T13:53:04.382 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:53:04.391 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-01-16T13:53:02.413046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219928,"kb_used_data":3688,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754792,"statfs":{"total":773094113280,"available":772868907008,"internally_reserved":0,"allocated":3776512,"data_stored":2132168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.878047"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540461+0000","last_change":"2024-01-16T13:52:57.540461+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540461+0000","last_undegraded":"2024-01-16T13:52:57.540461+0000","last_fullsized":"2024-01-16T13:52:57.540461+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541729+0000","last_change":"2024-01-16T13:52:57.541729+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541729+0000","last_undegraded":"2024-01-16T13:52:57.541729+0000","last_fullsized":"2024-01-16T13:52:57.541729+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541489+0000","last_change":"2024-01-16T13:52:57.541489+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541489+0000","last_undegraded":"2024-01-16T13:52:57.541489+0000","last_fullsized":"2024-01-16T13:52:57.541489+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538106+0000","last_change":"2024-01-16T13:52:57.538106+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538106+0000","last_undegraded":"2024-01-16T13:52:57.538106+0000","last_fullsized":"2024-01-16T13:52:57.538106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538125+0000","last_change":"2024-01-16T13:52:57.538125+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538125+0000","last_undegraded":"2024-01-16T13:52:57.538125+0000","last_fullsized":"2024-01-16T13:52:57.538125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542100+0000","last_change":"2024-01-16T13:52:57.542100+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542100+0000","last_undegraded":"2024-01-16T13:52:57.542100+0000","last_fullsized":"2024-01-16T13:52:57.542100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537122+0000","last_change":"2024-01-16T13:52:57.537122+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537122+0000","last_undegraded":"2024-01-16T13:52:57.537122+0000","last_fullsized":"2024-01-16T13:52:57.537122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541902+0000","last_change":"2024-01-16T13:52:57.541902+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541902+0000","last_undegraded":"2024-01-16T13:52:57.541902+0000","last_fullsized":"2024-01-16T13:52:57.541902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537171+0000","last_change":"2024-01-16T13:52:57.537171+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537171+0000","last_undegraded":"2024-01-16T13:52:57.537171+0000","last_fullsized":"2024-01-16T13:52:57.537171+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542291+0000","last_change":"2024-01-16T13:52:57.542291+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542291+0000","last_undegraded":"2024-01-16T13:52:57.542291+0000","last_fullsized":"2024-01-16T13:52:57.542291+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537449+0000","last_change":"2024-01-16T13:52:57.537449+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537449+0000","last_undegraded":"2024-01-16T13:52:57.537449+0000","last_fullsized":"2024-01-16T13:52:57.537449+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538504+0000","last_change":"2024-01-16T13:52:57.538504+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538504+0000","last_undegraded":"2024-01-16T13:52:57.538504+0000","last_fullsized":"2024-01-16T13:52:57.538504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541627+0000","last_change":"2024-01-16T13:52:57.541627+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541627+0000","last_undegraded":"2024-01-16T13:52:57.541627+0000","last_fullsized":"2024-01-16T13:52:57.541627+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.542716+0000","last_change":"2024-01-16T13:52:57.542716+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.542716+0000","last_undegraded":"2024-01-16T13:52:57.542716+0000","last_fullsized":"2024-01-16T13:52:57.542716+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538596+0000","last_change":"2024-01-16T13:52:57.538596+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538596+0000","last_undegraded":"2024-01-16T13:52:57.538596+0000","last_fullsized":"2024-01-16T13:52:57.538596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537813+0000","last_change":"2024-01-16T13:52:57.537813+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537813+0000","last_undegraded":"2024-01-16T13:52:57.537813+0000","last_fullsized":"2024-01-16T13:52:57.537813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537341+0000","last_change":"2024-01-16T13:52:57.537341+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537341+0000","last_undegraded":"2024-01-16T13:52:57.537341+0000","last_fullsized":"2024-01-16T13:52:57.537341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.538104+0000","last_change":"2024-01-16T13:52:57.538104+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.538104+0000","last_undegraded":"2024-01-16T13:52:57.538104+0000","last_fullsized":"2024-01-16T13:52:57.538104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540353+0000","last_change":"2024-01-16T13:52:57.540353+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540353+0000","last_undegraded":"2024-01-16T13:52:57.540353+0000","last_fullsized":"2024-01-16T13:52:57.540353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537698+0000","last_change":"2024-01-16T13:52:57.537698+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537698+0000","last_undegraded":"2024-01-16T13:52:57.537698+0000","last_fullsized":"2024-01-16T13:52:57.537698+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537078+0000","last_change":"2024-01-16T13:52:57.537078+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537078+0000","last_undegraded":"2024-01-16T13:52:57.537078+0000","last_fullsized":"2024-01-16T13:52:57.537078+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537842+0000","last_change":"2024-01-16T13:52:57.537842+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537842+0000","last_undegraded":"2024-01-16T13:52:57.537842+0000","last_fullsized":"2024-01-16T13:52:57.537842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537806+0000","last_change":"2024-01-16T13:52:57.537806+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537806+0000","last_undegraded":"2024-01-16T13:52:57.537806+0000","last_fullsized":"2024-01-16T13:52:57.537806+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:57.532220+0000","last_change":"2024-01-16T13:52:57.532220+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.532220+0000","last_undegraded":"2024-01-16T13:52:57.532220+0000","last_fullsized":"2024-01-16T13:52:57.532220+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537817+0000","last_change":"2024-01-16T13:52:57.537817+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537817+0000","last_undegraded":"2024-01-16T13:52:57.537817+0000","last_fullsized":"2024-01-16T13:52:57.537817+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.537104+0000","last_change":"2024-01-16T13:52:57.537104+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.537104+0000","last_undegraded":"2024-01-16T13:52:57.537104+0000","last_fullsized":"2024-01-16T13:52:57.537104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.541601+0000","last_change":"2024-01-16T13:52:57.541601+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.541601+0000","last_undegraded":"2024-01-16T13:52:57.541601+0000","last_fullsized":"2024-01-16T13:52:57.541601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-16T13:52:58.534818+0000","last_change":"2024-01-16T13:52:58.534818+0000","last_active":"2024-01-16T13:52:58.534818+0000","last_peered":"2024-01-16T13:52:58.534818+0000","last_clean":"2024-01-16T13:52:58.534818+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:58.534818+0000","last_undegraded":"2024-01-16T13:52:58.534818+0000","last_fullsized":"2024-01-16T13:52:58.534818+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-01-16T13:52:57.540422+0000","last_change":"2024-01-16T13:52:57.540422+0000","last_active":"2024-01-16T13:52:57.532220+0000","last_peered":"2024-01-16T13:52:57.532220+0000","last_clean":"2024-01-16T13:52:57.532220+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T13:52:57.540422+0000","last_undegraded":"2024-01-16T13:52:57.540422+0000","last_fullsized":"2024-01-16T13:52:57.540422+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-16T13:52:52.602144+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:52.602144+0000","last_peered":"2024-01-16T13:52:52.602144+0000","last_clean":"2024-01-16T13:52:52.602144+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:52.602144+0000","last_undegraded":"2024-01-16T13:52:52.602144+0000","last_fullsized":"2024-01-16T13:52:52.602144+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":917504,"data_stored":709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.52800000000000002}]},{"osd":3,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42199999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.628}]}]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.374,"5min":0.374,"15min":0.374},"last":0.34000000000000002}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"last":0.247},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.442}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.37},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.26000000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.32000000000000001}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.58699999999999997}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.48699999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.254},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.32700000000000001}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.372}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.38900000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41699999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.44700000000000001}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.55200000000000005}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.498},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":323584,"data_stored":118929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:53:04.395 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-01-16T13:53:04.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.395 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-01-16T13:53:04.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-01-16T13:53:04.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-01-16T13:53:04.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-01-16T13:53:04.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-01-16T13:53:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1f is not active+clean 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.540461+0000', 'last_change': '2024-01-16T13:52:57.540461+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.540461+0000', 'last_undegraded': '2024-01-16T13:52:57.540461+0000', 'last_fullsized': '2024-01-16T13:52:57.540461+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2024-01-16T13:53:04.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:57.532220+0000', 'last_change': '2024-01-16T13:52:57.532220+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.532220+0000', 'last_undegraded': '2024-01-16T13:52:57.532220+0000', 'last_fullsized': '2024-01-16T13:52:57.532220+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1d is not active+clean 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.541729+0000', 'last_change': '2024-01-16T13:52:57.541729+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.541729+0000', 'last_undegraded': '2024-01-16T13:52:57.541729+0000', 'last_fullsized': '2024-01-16T13:52:57.541729+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.541489+0000', 'last_change': '2024-01-16T13:52:57.541489+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.541489+0000', 'last_undegraded': '2024-01-16T13:52:57.541489+0000', 'last_fullsized': '2024-01-16T13:52:57.541489+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1b is not active+clean 2024-01-16T13:53:04.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.538106+0000', 'last_change': '2024-01-16T13:52:57.538106+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.538106+0000', 'last_undegraded': '2024-01-16T13:52:57.538106+0000', 'last_fullsized': '2024-01-16T13:52:57.538106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1a is not active+clean 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.538125+0000', 'last_change': '2024-01-16T13:52:57.538125+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.538125+0000', 'last_undegraded': '2024-01-16T13:52:57.538125+0000', 'last_fullsized': '2024-01-16T13:52:57.538125+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.19 is not active+clean 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.542100+0000', 'last_change': '2024-01-16T13:52:57.542100+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.542100+0000', 'last_undegraded': '2024-01-16T13:52:57.542100+0000', 'last_fullsized': '2024-01-16T13:52:57.542100+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.18 is not active+clean 2024-01-16T13:53:04.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537122+0000', 'last_change': '2024-01-16T13:52:57.537122+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537122+0000', 'last_undegraded': '2024-01-16T13:52:57.537122+0000', 'last_fullsized': '2024-01-16T13:52:57.537122+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.541902+0000', 'last_change': '2024-01-16T13:52:57.541902+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.541902+0000', 'last_undegraded': '2024-01-16T13:52:57.541902+0000', 'last_fullsized': '2024-01-16T13:52:57.541902+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.16 is not active+clean 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:57.532220+0000', 'last_change': '2024-01-16T13:52:57.532220+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.532220+0000', 'last_undegraded': '2024-01-16T13:52:57.532220+0000', 'last_fullsized': '2024-01-16T13:52:57.532220+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.15 is not active+clean 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537171+0000', 'last_change': '2024-01-16T13:52:57.537171+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537171+0000', 'last_undegraded': '2024-01-16T13:52:57.537171+0000', 'last_fullsized': '2024-01-16T13:52:57.537171+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.542291+0000', 'last_change': '2024-01-16T13:52:57.542291+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.542291+0000', 'last_undegraded': '2024-01-16T13:52:57.542291+0000', 'last_fullsized': '2024-01-16T13:52:57.542291+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.13 is not active+clean 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537449+0000', 'last_change': '2024-01-16T13:52:57.537449+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537449+0000', 'last_undegraded': '2024-01-16T13:52:57.537449+0000', 'last_fullsized': '2024-01-16T13:52:57.537449+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-01-16T13:53:04.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.12 is not active+clean 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.538504+0000', 'last_change': '2024-01-16T13:52:57.538504+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.538504+0000', 'last_undegraded': '2024-01-16T13:52:57.538504+0000', 'last_fullsized': '2024-01-16T13:52:57.538504+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.541627+0000', 'last_change': '2024-01-16T13:52:57.541627+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.541627+0000', 'last_undegraded': '2024-01-16T13:52:57.541627+0000', 'last_fullsized': '2024-01-16T13:52:57.541627+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.10 is not active+clean 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.542716+0000', 'last_change': '2024-01-16T13:52:57.542716+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.542716+0000', 'last_undegraded': '2024-01-16T13:52:57.542716+0000', 'last_fullsized': '2024-01-16T13:52:57.542716+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-01-16T13:53:04.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.f is not active+clean 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.538596+0000', 'last_change': '2024-01-16T13:52:57.538596+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.538596+0000', 'last_undegraded': '2024-01-16T13:52:57.538596+0000', 'last_fullsized': '2024-01-16T13:52:57.538596+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.e is not active+clean 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537813+0000', 'last_change': '2024-01-16T13:52:57.537813+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537813+0000', 'last_undegraded': '2024-01-16T13:52:57.537813+0000', 'last_fullsized': '2024-01-16T13:52:57.537813+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.d is not active+clean 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537341+0000', 'last_change': '2024-01-16T13:52:57.537341+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537341+0000', 'last_undegraded': '2024-01-16T13:52:57.537341+0000', 'last_fullsized': '2024-01-16T13:52:57.537341+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-01-16T13:53:04.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.c is not active+clean 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.538104+0000', 'last_change': '2024-01-16T13:52:57.538104+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.538104+0000', 'last_undegraded': '2024-01-16T13:52:57.538104+0000', 'last_fullsized': '2024-01-16T13:52:57.538104+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.b is not active+clean 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:57.532220+0000', 'last_change': '2024-01-16T13:52:57.532220+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.532220+0000', 'last_undegraded': '2024-01-16T13:52:57.532220+0000', 'last_fullsized': '2024-01-16T13:52:57.532220+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.a is not active+clean 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.540353+0000', 'last_change': '2024-01-16T13:52:57.540353+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.540353+0000', 'last_undegraded': '2024-01-16T13:52:57.540353+0000', 'last_fullsized': '2024-01-16T13:52:57.540353+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T13:53:04.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.9 is not active+clean 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537698+0000', 'last_change': '2024-01-16T13:52:57.537698+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537698+0000', 'last_undegraded': '2024-01-16T13:52:57.537698+0000', 'last_fullsized': '2024-01-16T13:52:57.537698+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.8 is not active+clean 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537078+0000', 'last_change': '2024-01-16T13:52:57.537078+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537078+0000', 'last_undegraded': '2024-01-16T13:52:57.537078+0000', 'last_fullsized': '2024-01-16T13:52:57.537078+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.7 is not active+clean 2024-01-16T13:53:04.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537842+0000', 'last_change': '2024-01-16T13:52:57.537842+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537842+0000', 'last_undegraded': '2024-01-16T13:52:57.537842+0000', 'last_fullsized': '2024-01-16T13:52:57.537842+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.6 is not active+clean 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:57.532220+0000', 'last_change': '2024-01-16T13:52:57.532220+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.532220+0000', 'last_undegraded': '2024-01-16T13:52:57.532220+0000', 'last_fullsized': '2024-01-16T13:52:57.532220+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537806+0000', 'last_change': '2024-01-16T13:52:57.537806+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537806+0000', 'last_undegraded': '2024-01-16T13:52:57.537806+0000', 'last_fullsized': '2024-01-16T13:52:57.537806+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3 is not active+clean 2024-01-16T13:53:04.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:57.532220+0000', 'last_change': '2024-01-16T13:52:57.532220+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.532220+0000', 'last_undegraded': '2024-01-16T13:52:57.532220+0000', 'last_fullsized': '2024-01-16T13:52:57.532220+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2 is not active+clean 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537817+0000', 'last_change': '2024-01-16T13:52:57.537817+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537817+0000', 'last_undegraded': '2024-01-16T13:52:57.537817+0000', 'last_fullsized': '2024-01-16T13:52:57.537817+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1 is not active+clean 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.537104+0000', 'last_change': '2024-01-16T13:52:57.537104+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.537104+0000', 'last_undegraded': '2024-01-16T13:52:57.537104+0000', 'last_fullsized': '2024-01-16T13:52:57.537104+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.0 is not active+clean 2024-01-16T13:53:04.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.541601+0000', 'last_change': '2024-01-16T13:52:57.541601+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.541601+0000', 'last_undegraded': '2024-01-16T13:52:57.541601+0000', 'last_fullsized': '2024-01-16T13:52:57.541601+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-16T13:53:04.412 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-01-16T13:53:04.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-16T13:52:58.534818+0000', 'last_change': '2024-01-16T13:52:58.534818+0000', 'last_active': '2024-01-16T13:52:58.534818+0000', 'last_peered': '2024-01-16T13:52:58.534818+0000', 'last_clean': '2024-01-16T13:52:58.534818+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:58.534818+0000', 'last_undegraded': '2024-01-16T13:52:58.534818+0000', 'last_fullsized': '2024-01-16T13:52:58.534818+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:58.534818+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-16T13:53:04.412 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.5 is not active+clean 2024-01-16T13:53:04.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-01-16T13:52:57.540422+0000', 'last_change': '2024-01-16T13:52:57.540422+0000', 'last_active': '2024-01-16T13:52:57.532220+0000', 'last_peered': '2024-01-16T13:52:57.532220+0000', 'last_clean': '2024-01-16T13:52:57.532220+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T13:52:57.540422+0000', 'last_undegraded': '2024-01-16T13:52:57.540422+0000', 'last_fullsized': '2024-01-16T13:52:57.540422+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'last_clean_scrub_stamp': '2024-01-16T13:52:57.532220+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T13:53:04.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-16T13:53:04.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.570+0000 7f5c39b88700 1 Processor -- start 2024-01-16T13:53:04.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.570+0000 7f5c39b88700 1 -- start start 2024-01-16T13:53:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c39b88700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5c3406c710 0x7f5c340b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c39b88700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c340ae430 con 0x7f5c3415e080 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c39b88700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c340ae710 con 0x7f5c340af4d0 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c39b88700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c340af0b0 con 0x7f5c3406c710 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c337fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f5c3415e080 0x7f5c3406c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:45248/0 (socket says 172.21.15.113:45248) 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c337fe700 1 -- 172.21.15.113:0/2092662919 learned_addr learned my addr 172.21.15.113:0/2092662919 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c32ffd700 1 --2- 172.21.15.113:0/2092662919 >> v2:172.21.15.124:3300/0 conn(0x7f5c3406c710 0x7f5c340b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480313038 0 0) 0x7f5c340ae430 con 0x7f5c3415e080 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.571+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c200035b0 con 0x7f5c3415e080 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863921149 0 0) 0x7f5c200035b0 con 0x7f5c3415e080 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 >> v2:172.21.15.124:3300/0 conn(0x7f5c3406c710 msgr2=0x7f5c340b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 --2- 172.21.15.113:0/2092662919 >> v2:172.21.15.124:3300/0 conn(0x7f5c3406c710 0x7f5c340b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 >> v1:172.21.15.124:6789/0 conn(0x7f5c340af4d0 legacy=0x7f5c3406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c340ae9f0 con 0x7f5c3415e080 2024-01-16T13:53:04.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f5c1c001670 con 0x7f5c3415e080 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 >> v1:172.21.15.113:6789/0 conn(0x7f5c3415e080 legacy=0x7f5c3406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c20003a00 0x7f5c20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c20004850 0x7f5c20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c2000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c340af0b0 con 0x7f5c20005210 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2000c0b0 con 0x7f5c20004850 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2000c270 con 0x7f5c20003a00 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c2000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c337fe700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c20003a00 0x7f5c20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c32ffd700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c20004850 0x7f5c20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 -- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c20003a00 msgr2=0x7f5c20003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c20003a00 0x7f5c20003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 -- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c20004850 msgr2=0x7f5c20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c20004850 0x7f5c20004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 -- 172.21.15.113:0/2092662919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200035b0 con 0x7f5c20005210 2024-01-16T13:53:04.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 -- 172.21.15.113:0/2092662919 reap_dead start 2024-01-16T13:53:04.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.572+0000 7f5c33fff700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c2000ba10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00c190 tx=0x7f5c20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf5f00742ef0f4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.574+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c001db0 con 0x7f5c20005210 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.574+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c00ccf0 con 0x7f5c20005210 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.574+0000 7f5c327fc700 1 -- 172.21.15.113:0/2092662919 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c003c40 con 0x7f5c20005210 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.574+0000 7f5c39b88700 1 -- 172.21.15.113:0/2092662919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 msgr2=0x7f5c2000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.574+0000 7f5c39b88700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c2000ba10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00c190 tx=0x7f5c20005880 comp rx=0 tx=0).stop 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.574+0000 7f5c39b88700 1 -- 172.21.15.113:0/2092662919 shutdown_connections 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.575+0000 7f5c39b88700 1 --2- 172.21.15.113:0/2092662919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c2000ba10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.575+0000 7f5c39b88700 1 -- 172.21.15.113:0/2092662919 >> 172.21.15.113:0/2092662919 conn(0x7f5c34071b90 msgr2=0x7f5c34071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.575+0000 7f5c39b88700 1 -- 172.21.15.113:0/2092662919 shutdown_connections 2024-01-16T13:53:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.575+0000 7f5c39b88700 1 -- 172.21.15.113:0/2092662919 wait complete. 2024-01-16T13:53:04.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.576+0000 7f5c39b88700 1 Processor -- start 2024-01-16T13:53:04.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.576+0000 7f5c39b88700 1 -- start start 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c39b88700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c3415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c39b88700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c341562e0 0x7f5c34142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c39b88700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c34150e20 0x7f5c34151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c39b88700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34156b60 con 0x7f5c20005210 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c39b88700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c341517a0 con 0x7f5c34150e20 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c39b88700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3415fb20 con 0x7f5c341562e0 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c3415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c3415b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50020/0 (socket says 172.21.15.113:50020) 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 -- 172.21.15.113:0/3419254258 learned_addr learned my addr 172.21.15.113:0/3419254258 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c33fff700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c34150e20 0x7f5c34151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c32ffd700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c341562e0 0x7f5c34142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 -- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c341562e0 msgr2=0x7f5c34142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c341562e0 0x7f5c34142350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 -- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c34150e20 msgr2=0x7f5c34151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c34150e20 0x7f5c34151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 -- 172.21.15.113:0/3419254258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3415fdc0 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c337fe700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c3415b1c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0072c0 tx=0x7f5c1c0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.577+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00f6c0 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.578+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c00f840 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.578+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3415b700 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.578+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00fbf0 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.578+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c34160060 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.578+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c340753f0 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.579+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5c1c00fd50 con 0x7f5c20005210 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.579+0000 7f5c2bfff700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5c1406b350 0x7f5c1406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.579+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c1c089c00 con 0x7f5c20005210 2024-01-16T13:53:04.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.579+0000 7f5c32ffd700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5c1406b350 0x7f5c1406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.580+0000 7f5c32ffd700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5c1406b350 0x7f5c1406d810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c3415a760 tx=0x7f5c24009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.584+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c1c0549f0 con 0x7f5c20005210 2024-01-16T13:53:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.853+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5c34067f60 con 0x7f5c20005210 2024-01-16T13:53:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.853+0000 7f5c2bfff700 1 -- 172.21.15.113:0/3419254258 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1559 (secure 0 0 0) 0x7f5c1c058010 con 0x7f5c20005210 2024-01-16T13:53:04.835 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:04.835 INFO:teuthology.orchestra.run.smithi113.stdout:{"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":78,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1705413116,"num_in_osds":8,"osd_in_since":1705413109,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":225611776,"bytes_avail":772868501504,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5067}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-01-16T13:52:58.412474+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-01-16T13:53:04.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5c1406b350 msgr2=0x7f5c1406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5c1406b350 0x7f5c1406d810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c3415a760 tx=0x7f5c24009210 comp rx=0 tx=0).stop 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 msgr2=0x7f5c3415b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c3415b1c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0072c0 tx=0x7f5c1c0075d0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 shutdown_connections 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5c1406b350 0x7f5c1406d810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c34150e20 0x7f5c34151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c20005210 0x7f5c3415b1c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 --2- 172.21.15.113:0/3419254258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c341562e0 0x7f5c34142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 >> 172.21.15.113:0/3419254258 conn(0x7f5c34071b90 msgr2=0x7f5c340b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 shutdown_connections 2024-01-16T13:53:04.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.855+0000 7f5c39b88700 1 -- 172.21.15.113:0/3419254258 wait complete. 2024-01-16T13:53:04.849 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-16T13:53:04.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:53:04.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.988+0000 7f8924374700 1 Processor -- start 2024-01-16T13:53:04.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.988+0000 7f8924374700 1 -- start start 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8924374700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f891c15bed0 0x7f891c0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8924374700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 0x7f891c0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8924374700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f891c0d1a50 0x7f891c0d1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8924374700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891c15c2b0 con 0x7f891c0d1a50 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8924374700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891c0d0770 con 0x7f891c15bed0 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8924374700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0cd100 con 0x7f891c1603c0 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922911700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f891c0d1a50 0x7f891c0d1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922911700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f891c0d1a50 0x7f891c0d1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50022/0 (socket says 172.21.15.113:50022) 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922911700 1 -- 172.21.15.113:0/2224401028 learned_addr learned my addr 172.21.15.113:0/2224401028 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922911700 1 -- 172.21.15.113:0/2224401028 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 msgr2=0x7f891c0d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922911700 1 --2- 172.21.15.113:0/2224401028 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 0x7f891c0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922911700 1 -- 172.21.15.113:0/2224401028 >> v2:172.21.15.124:3300/0 conn(0x7f891c15bed0 msgr2=0x7f891c0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.989+0000 7f8922110700 1 --2- 172.21.15.113:0/2224401028 >> v2:172.21.15.124:3300/0 conn(0x7f891c15bed0 0x7f891c0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8922911700 1 --2- 172.21.15.113:0/2224401028 >> v2:172.21.15.124:3300/0 conn(0x7f891c15bed0 0x7f891c0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8922911700 1 -- 172.21.15.113:0/2224401028 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c0d2420 con 0x7f891c0d1a50 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8922911700 1 --2- 172.21.15.113:0/2224401028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c0d1a50 0x7f891c0d1ee0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f891c09b2a0 tx=0x7f891400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1092f231fc9f006 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f892110e700 1 -- 172.21.15.113:0/2224401028 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891400e070 con 0x7f891c0d1a50 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f892110e700 1 -- 172.21.15.113:0/2224401028 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891400b9c0 con 0x7f891c0d1a50 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f892110e700 1 -- 172.21.15.113:0/2224401028 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891400bd70 con 0x7f891c0d1a50 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 -- 172.21.15.113:0/2224401028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c0d1a50 msgr2=0x7f891c0d1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 --2- 172.21.15.113:0/2224401028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c0d1a50 0x7f891c0d1ee0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f891c09b2a0 tx=0x7f891400b330 comp rx=0 tx=0).stop 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 -- 172.21.15.113:0/2224401028 shutdown_connections 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 --2- 172.21.15.113:0/2224401028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c0d1a50 0x7f891c0d1ee0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 --2- 172.21.15.113:0/2224401028 >> v2:172.21.15.124:3300/0 conn(0x7f891c15bed0 0x7f891c0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 --2- 172.21.15.113:0/2224401028 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 0x7f891c0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 -- 172.21.15.113:0/2224401028 >> 172.21.15.113:0/2224401028 conn(0x7f891c073130 msgr2=0x7f891c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 -- 172.21.15.113:0/2224401028 shutdown_connections 2024-01-16T13:53:04.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.990+0000 7f8924374700 1 -- 172.21.15.113:0/2224401028 wait complete. 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.992+0000 7f8924374700 1 Processor -- start 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.992+0000 7f8924374700 1 -- start start 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8924374700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f891c0d1a50 0x7f891c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8924374700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 0x7f891c1f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8924374700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 0x7f891c1f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8924374700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0b5a10 con 0x7f891c15bed0 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8924374700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0b5cf0 con 0x7f891c0d1a50 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8924374700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0b5fd0 con 0x7f891c1603c0 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 0x7f891c1f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 0x7f891c1f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50036/0 (socket says 172.21.15.113:50036) 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 -- 172.21.15.113:0/1836052591 learned_addr learned my addr 172.21.15.113:0/1836052591 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f8922110700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f891c0d1a50 0x7f891c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 -- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 msgr2=0x7f891c1f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 0x7f891c1f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 -- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f891c0d1a50 msgr2=0x7f891c0b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f891c0d1a50 0x7f891c0b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 -- 172.21.15.113:0/1836052591 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8914009b30 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.993+0000 7f892190f700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 0x7f891c1f4ca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f891800b530 tx=0x7f891800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.994+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918011830 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.994+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89180119b0 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.994+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918011d60 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.994+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891c0b62b0 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.994+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891c0c85a0 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.995+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891c0b67d0 con 0x7f891c15bed0 2024-01-16T13:53:04.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.995+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f891801a400 con 0x7f891c15bed0 2024-01-16T13:53:04.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.995+0000 7f8912ffd700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f890006b350 0x7f890006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:04.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.996+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8918089c80 con 0x7f891c15bed0 2024-01-16T13:53:04.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.996+0000 7f8922110700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f890006b350 0x7f890006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:04.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:04.996+0000 7f8922110700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f890006b350 0x7f890006d810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f890c008850 tx=0x7f890c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:04.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.000+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f891c0b67d0 con 0x7f891c15bed0 2024-01-16T13:53:05.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.184+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f891c0b5e60 con 0x7f890006b350 2024-01-16T13:53:05.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.192+0000 7f8912ffd700 1 -- 172.21.15.113:0/1836052591 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289576 (secure 0 0 0) 0x7f891c0b5e60 con 0x7f890006b350 2024-01-16T13:53:05.177 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f890006b350 msgr2=0x7f890006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f890006b350 0x7f890006d810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f890c008850 tx=0x7f890c008040 comp rx=0 tx=0).stop 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 msgr2=0x7f891c1f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 0x7f891c1f4ca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f891800b530 tx=0x7f891800b8f0 comp rx=0 tx=0).stop 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 shutdown_connections 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f890006b350 0x7f890006d810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f891c1603c0 0x7f891c1f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f891c0d1a50 0x7f891c0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 --2- 172.21.15.113:0/1836052591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f891c15bed0 0x7f891c1f4ca0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 >> 172.21.15.113:0/1836052591 conn(0x7f891c073130 msgr2=0x7f891c0b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 shutdown_connections 2024-01-16T13:53:05.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:05.195+0000 7f8924374700 1 -- 172.21.15.113:0/1836052591 wait complete. 2024-01-16T13:53:05.180 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:53:05.189 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-01-16T13:53:04.413495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220324,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754396,"statfs":{"total":773094113280,"available":772868501504,"internally_reserved":0,"allocated":4235264,"data_stored":2338260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001139"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792374+0000","last_change":"2024-01-16T13:52:59.573148+0000","last_active":"2024-01-16T13:53:00.792374+0000","last_peered":"2024-01-16T13:53:00.792374+0000","last_clean":"2024-01-16T13:53:00.792374+0000","last_became_active":"2024-01-16T13:52:59.568465+0000","last_became_peered":"2024-01-16T13:52:59.568465+0000","last_unstale":"2024-01-16T13:53:00.792374+0000","last_undegraded":"2024-01-16T13:53:00.792374+0000","last_fullsized":"2024-01-16T13:53:00.792374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:30.878710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609874+0000","last_change":"2024-01-16T13:52:59.568178+0000","last_active":"2024-01-16T13:53:00.609874+0000","last_peered":"2024-01-16T13:53:00.609874+0000","last_clean":"2024-01-16T13:53:00.609874+0000","last_became_active":"2024-01-16T13:52:59.567905+0000","last_became_peered":"2024-01-16T13:52:59.567905+0000","last_unstale":"2024-01-16T13:53:00.609874+0000","last_undegraded":"2024-01-16T13:53:00.609874+0000","last_fullsized":"2024-01-16T13:53:00.609874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:34.906681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563165+0000","last_change":"2024-01-16T13:52:59.573614+0000","last_active":"2024-01-16T13:53:00.563165+0000","last_peered":"2024-01-16T13:53:00.563165+0000","last_clean":"2024-01-16T13:53:00.563165+0000","last_became_active":"2024-01-16T13:52:59.573292+0000","last_became_peered":"2024-01-16T13:52:59.573292+0000","last_unstale":"2024-01-16T13:53:00.563165+0000","last_undegraded":"2024-01-16T13:53:00.563165+0000","last_fullsized":"2024-01-16T13:53:00.563165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:12:41.316957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.559981+0000","last_change":"2024-01-16T13:52:59.569475+0000","last_active":"2024-01-16T13:53:00.559981+0000","last_peered":"2024-01-16T13:53:00.559981+0000","last_clean":"2024-01-16T13:53:00.559981+0000","last_became_active":"2024-01-16T13:52:59.569161+0000","last_became_peered":"2024-01-16T13:52:59.569161+0000","last_unstale":"2024-01-16T13:53:00.559981+0000","last_undegraded":"2024-01-16T13:53:00.559981+0000","last_fullsized":"2024-01-16T13:53:00.559981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:53.015251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791137+0000","last_change":"2024-01-16T13:52:59.564689+0000","last_active":"2024-01-16T13:53:00.791137+0000","last_peered":"2024-01-16T13:53:00.791137+0000","last_clean":"2024-01-16T13:53:00.791137+0000","last_became_active":"2024-01-16T13:52:59.564414+0000","last_became_peered":"2024-01-16T13:52:59.564414+0000","last_unstale":"2024-01-16T13:53:00.791137+0000","last_undegraded":"2024-01-16T13:53:00.791137+0000","last_fullsized":"2024-01-16T13:53:00.791137+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:15.726165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609629+0000","last_change":"2024-01-16T13:52:59.563734+0000","last_active":"2024-01-16T13:53:00.609629+0000","last_peered":"2024-01-16T13:53:00.609629+0000","last_clean":"2024-01-16T13:53:00.609629+0000","last_became_active":"2024-01-16T13:52:59.561315+0000","last_became_peered":"2024-01-16T13:52:59.561315+0000","last_unstale":"2024-01-16T13:53:00.609629+0000","last_undegraded":"2024-01-16T13:53:00.609629+0000","last_fullsized":"2024-01-16T13:53:00.609629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:12:41.316957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792646+0000","last_change":"2024-01-16T13:52:59.567372+0000","last_active":"2024-01-16T13:53:00.792646+0000","last_peered":"2024-01-16T13:53:00.792646+0000","last_clean":"2024-01-16T13:53:00.792646+0000","last_became_active":"2024-01-16T13:52:59.566993+0000","last_became_peered":"2024-01-16T13:52:59.566993+0000","last_unstale":"2024-01-16T13:53:00.792646+0000","last_undegraded":"2024-01-16T13:53:00.792646+0000","last_fullsized":"2024-01-16T13:53:00.792646+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:12:41.316957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792382+0000","last_change":"2024-01-16T13:52:59.566131+0000","last_active":"2024-01-16T13:53:00.792382+0000","last_peered":"2024-01-16T13:53:00.792382+0000","last_clean":"2024-01-16T13:53:00.792382+0000","last_became_active":"2024-01-16T13:52:59.565866+0000","last_became_peered":"2024-01-16T13:52:59.565866+0000","last_unstale":"2024-01-16T13:53:00.792382+0000","last_undegraded":"2024-01-16T13:53:00.792382+0000","last_fullsized":"2024-01-16T13:53:00.792382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:32:24.125629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792912+0000","last_change":"2024-01-16T13:52:59.570127+0000","last_active":"2024-01-16T13:53:00.792912+0000","last_peered":"2024-01-16T13:53:00.792912+0000","last_clean":"2024-01-16T13:53:00.792912+0000","last_became_active":"2024-01-16T13:52:59.569670+0000","last_became_peered":"2024-01-16T13:52:59.569670+0000","last_unstale":"2024-01-16T13:53:00.792912+0000","last_undegraded":"2024-01-16T13:53:00.792912+0000","last_fullsized":"2024-01-16T13:53:00.792912+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:20.325793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560200+0000","last_change":"2024-01-16T13:52:59.566004+0000","last_active":"2024-01-16T13:53:00.560200+0000","last_peered":"2024-01-16T13:53:00.560200+0000","last_clean":"2024-01-16T13:53:00.560200+0000","last_became_active":"2024-01-16T13:52:59.565695+0000","last_became_peered":"2024-01-16T13:52:59.565695+0000","last_unstale":"2024-01-16T13:53:00.560200+0000","last_undegraded":"2024-01-16T13:53:00.560200+0000","last_fullsized":"2024-01-16T13:53:00.560200+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:28.921798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793129+0000","last_change":"2024-01-16T13:52:59.568183+0000","last_active":"2024-01-16T13:53:00.793129+0000","last_peered":"2024-01-16T13:53:00.793129+0000","last_clean":"2024-01-16T13:53:00.793129+0000","last_became_active":"2024-01-16T13:52:59.567833+0000","last_became_peered":"2024-01-16T13:52:59.567833+0000","last_unstale":"2024-01-16T13:53:00.793129+0000","last_undegraded":"2024-01-16T13:53:00.793129+0000","last_fullsized":"2024-01-16T13:53:00.793129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:08.471154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563286+0000","last_change":"2024-01-16T13:52:59.570264+0000","last_active":"2024-01-16T13:53:00.563286+0000","last_peered":"2024-01-16T13:53:00.563286+0000","last_clean":"2024-01-16T13:53:00.563286+0000","last_became_active":"2024-01-16T13:52:59.569849+0000","last_became_peered":"2024-01-16T13:52:59.569849+0000","last_unstale":"2024-01-16T13:53:00.563286+0000","last_undegraded":"2024-01-16T13:53:00.563286+0000","last_fullsized":"2024-01-16T13:53:00.563286+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:25.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791268+0000","last_change":"2024-01-16T13:52:59.567270+0000","last_active":"2024-01-16T13:53:00.791268+0000","last_peered":"2024-01-16T13:53:00.791268+0000","last_clean":"2024-01-16T13:53:00.791268+0000","last_became_active":"2024-01-16T13:52:59.567003+0000","last_became_peered":"2024-01-16T13:52:59.567003+0000","last_unstale":"2024-01-16T13:53:00.791268+0000","last_undegraded":"2024-01-16T13:53:00.791268+0000","last_fullsized":"2024-01-16T13:53:00.791268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:15.142935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609550+0000","last_change":"2024-01-16T13:52:59.568677+0000","last_active":"2024-01-16T13:53:00.609550+0000","last_peered":"2024-01-16T13:53:00.609550+0000","last_clean":"2024-01-16T13:53:00.609550+0000","last_became_active":"2024-01-16T13:52:59.568410+0000","last_became_peered":"2024-01-16T13:52:59.568410+0000","last_unstale":"2024-01-16T13:53:00.609550+0000","last_undegraded":"2024-01-16T13:53:00.609550+0000","last_fullsized":"2024-01-16T13:53:00.609550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:21.972684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563074+0000","last_change":"2024-01-16T13:52:59.563121+0000","last_active":"2024-01-16T13:53:00.563074+0000","last_peered":"2024-01-16T13:53:00.563074+0000","last_clean":"2024-01-16T13:53:00.563074+0000","last_became_active":"2024-01-16T13:52:59.562812+0000","last_became_peered":"2024-01-16T13:52:59.562812+0000","last_unstale":"2024-01-16T13:53:00.563074+0000","last_undegraded":"2024-01-16T13:53:00.563074+0000","last_fullsized":"2024-01-16T13:53:00.563074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:18:18.563379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793121+0000","last_change":"2024-01-16T13:52:59.565538+0000","last_active":"2024-01-16T13:53:00.793121+0000","last_peered":"2024-01-16T13:53:00.793121+0000","last_clean":"2024-01-16T13:53:00.793121+0000","last_became_active":"2024-01-16T13:52:59.563776+0000","last_became_peered":"2024-01-16T13:52:59.563776+0000","last_unstale":"2024-01-16T13:53:00.793121+0000","last_undegraded":"2024-01-16T13:53:00.793121+0000","last_fullsized":"2024-01-16T13:53:00.793121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:34.906681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792450+0000","last_change":"2024-01-16T13:52:59.564678+0000","last_active":"2024-01-16T13:53:00.792450+0000","last_peered":"2024-01-16T13:53:00.792450+0000","last_clean":"2024-01-16T13:53:00.792450+0000","last_became_active":"2024-01-16T13:52:59.564382+0000","last_became_peered":"2024-01-16T13:52:59.564382+0000","last_unstale":"2024-01-16T13:53:00.792450+0000","last_undegraded":"2024-01-16T13:53:00.792450+0000","last_fullsized":"2024-01-16T13:53:00.792450+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:18:18.563379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609973+0000","last_change":"2024-01-16T13:52:59.570828+0000","last_active":"2024-01-16T13:53:00.609973+0000","last_peered":"2024-01-16T13:53:00.609973+0000","last_clean":"2024-01-16T13:53:00.609973+0000","last_became_active":"2024-01-16T13:52:59.570282+0000","last_became_peered":"2024-01-16T13:52:59.570282+0000","last_unstale":"2024-01-16T13:53:00.609973+0000","last_undegraded":"2024-01-16T13:53:00.609973+0000","last_fullsized":"2024-01-16T13:53:00.609973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:32.369974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609806+0000","last_change":"2024-01-16T13:52:59.570661+0000","last_active":"2024-01-16T13:53:00.609806+0000","last_peered":"2024-01-16T13:53:00.609806+0000","last_clean":"2024-01-16T13:53:00.609806+0000","last_became_active":"2024-01-16T13:52:59.570305+0000","last_became_peered":"2024-01-16T13:52:59.570305+0000","last_unstale":"2024-01-16T13:53:00.609806+0000","last_undegraded":"2024-01-16T13:53:00.609806+0000","last_fullsized":"2024-01-16T13:53:00.609806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:08.471154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791192+0000","last_change":"2024-01-16T13:52:59.567138+0000","last_active":"2024-01-16T13:53:00.791192+0000","last_peered":"2024-01-16T13:53:00.791192+0000","last_clean":"2024-01-16T13:53:00.791192+0000","last_became_active":"2024-01-16T13:52:59.566826+0000","last_became_peered":"2024-01-16T13:52:59.566826+0000","last_unstale":"2024-01-16T13:53:00.791192+0000","last_undegraded":"2024-01-16T13:53:00.791192+0000","last_fullsized":"2024-01-16T13:53:00.791192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:07.463675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792293+0000","last_change":"2024-01-16T13:52:59.569169+0000","last_active":"2024-01-16T13:53:00.792293+0000","last_peered":"2024-01-16T13:53:00.792293+0000","last_clean":"2024-01-16T13:53:00.792293+0000","last_became_active":"2024-01-16T13:52:59.568908+0000","last_became_peered":"2024-01-16T13:52:59.568908+0000","last_unstale":"2024-01-16T13:53:00.792293+0000","last_undegraded":"2024-01-16T13:53:00.792293+0000","last_fullsized":"2024-01-16T13:53:00.792293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:14.688323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793023+0000","last_change":"2024-01-16T13:52:59.554262+0000","last_active":"2024-01-16T13:53:00.793023+0000","last_peered":"2024-01-16T13:53:00.793023+0000","last_clean":"2024-01-16T13:53:00.793023+0000","last_became_active":"2024-01-16T13:52:59.553892+0000","last_became_peered":"2024-01-16T13:52:59.553892+0000","last_unstale":"2024-01-16T13:53:00.793023+0000","last_undegraded":"2024-01-16T13:53:00.793023+0000","last_fullsized":"2024-01-16T13:53:00.793023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:00.590118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562661+0000","last_change":"2024-01-16T13:52:59.564937+0000","last_active":"2024-01-16T13:53:00.562661+0000","last_peered":"2024-01-16T13:53:00.562661+0000","last_clean":"2024-01-16T13:53:00.562661+0000","last_became_active":"2024-01-16T13:52:59.564618+0000","last_became_peered":"2024-01-16T13:52:59.564618+0000","last_unstale":"2024-01-16T13:53:00.562661+0000","last_undegraded":"2024-01-16T13:53:00.562661+0000","last_fullsized":"2024-01-16T13:53:00.562661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:50:00.627242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562884+0000","last_change":"2024-01-16T13:52:59.565960+0000","last_active":"2024-01-16T13:53:00.562884+0000","last_peered":"2024-01-16T13:53:00.562884+0000","last_clean":"2024-01-16T13:53:00.562884+0000","last_became_active":"2024-01-16T13:52:59.565667+0000","last_became_peered":"2024-01-16T13:52:59.565667+0000","last_unstale":"2024-01-16T13:53:00.562884+0000","last_undegraded":"2024-01-16T13:53:00.562884+0000","last_fullsized":"2024-01-16T13:53:00.562884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:32:24.125629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560360+0000","last_change":"2024-01-16T13:52:59.565505+0000","last_active":"2024-01-16T13:53:00.560360+0000","last_peered":"2024-01-16T13:53:00.560360+0000","last_clean":"2024-01-16T13:53:00.560360+0000","last_became_active":"2024-01-16T13:52:59.565212+0000","last_became_peered":"2024-01-16T13:52:59.565212+0000","last_unstale":"2024-01-16T13:53:00.560360+0000","last_undegraded":"2024-01-16T13:53:00.560360+0000","last_fullsized":"2024-01-16T13:53:00.560360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:34.336351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609949+0000","last_change":"2024-01-16T13:52:59.563999+0000","last_active":"2024-01-16T13:53:00.609949+0000","last_peered":"2024-01-16T13:53:00.609949+0000","last_clean":"2024-01-16T13:53:00.609949+0000","last_became_active":"2024-01-16T13:52:59.563738+0000","last_became_peered":"2024-01-16T13:52:59.563738+0000","last_unstale":"2024-01-16T13:53:00.609949+0000","last_undegraded":"2024-01-16T13:53:00.609949+0000","last_fullsized":"2024-01-16T13:53:00.609949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:34.382768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560375+0000","last_change":"2024-01-16T13:52:59.567123+0000","last_active":"2024-01-16T13:53:00.560375+0000","last_peered":"2024-01-16T13:53:00.560375+0000","last_clean":"2024-01-16T13:53:00.560375+0000","last_became_active":"2024-01-16T13:52:59.566557+0000","last_became_peered":"2024-01-16T13:52:59.566557+0000","last_unstale":"2024-01-16T13:53:00.560375+0000","last_undegraded":"2024-01-16T13:53:00.560375+0000","last_fullsized":"2024-01-16T13:53:00.560375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:53.788479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563411+0000","last_change":"2024-01-16T13:52:59.565257+0000","last_active":"2024-01-16T13:53:00.563411+0000","last_peered":"2024-01-16T13:53:00.563411+0000","last_clean":"2024-01-16T13:53:00.563411+0000","last_became_active":"2024-01-16T13:52:59.564963+0000","last_became_peered":"2024-01-16T13:52:59.564963+0000","last_unstale":"2024-01-16T13:53:00.563411+0000","last_undegraded":"2024-01-16T13:53:00.563411+0000","last_fullsized":"2024-01-16T13:53:00.563411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:34.906681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792449+0000","last_change":"2024-01-16T13:52:59.554147+0000","last_active":"2024-01-16T13:53:00.792449+0000","last_peered":"2024-01-16T13:53:00.792449+0000","last_clean":"2024-01-16T13:53:00.792449+0000","last_became_active":"2024-01-16T13:52:59.553868+0000","last_became_peered":"2024-01-16T13:52:59.553868+0000","last_unstale":"2024-01-16T13:53:00.792449+0000","last_undegraded":"2024-01-16T13:53:00.792449+0000","last_fullsized":"2024-01-16T13:53:00.792449+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:27.471640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792170+0000","last_change":"2024-01-16T13:52:59.567154+0000","last_active":"2024-01-16T13:53:00.792170+0000","last_peered":"2024-01-16T13:53:00.792170+0000","last_clean":"2024-01-16T13:53:00.792170+0000","last_became_active":"2024-01-16T13:52:59.566871+0000","last_became_peered":"2024-01-16T13:52:59.566871+0000","last_unstale":"2024-01-16T13:53:00.792170+0000","last_undegraded":"2024-01-16T13:53:00.792170+0000","last_fullsized":"2024-01-16T13:53:00.792170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:02.720813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793277+0000","last_change":"2024-01-16T13:52:59.571801+0000","last_active":"2024-01-16T13:53:00.793277+0000","last_peered":"2024-01-16T13:53:00.793277+0000","last_clean":"2024-01-16T13:53:00.793277+0000","last_became_active":"2024-01-16T13:52:59.571468+0000","last_became_peered":"2024-01-16T13:52:59.571468+0000","last_unstale":"2024-01-16T13:53:00.793277+0000","last_undegraded":"2024-01-16T13:53:00.793277+0000","last_fullsized":"2024-01-16T13:53:00.793277+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:50:00.627242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792792+0000","last_change":"2024-01-16T13:52:59.565812+0000","last_active":"2024-01-16T13:53:00.792792+0000","last_peered":"2024-01-16T13:53:00.792792+0000","last_clean":"2024-01-16T13:53:00.792792+0000","last_became_active":"2024-01-16T13:52:59.565543+0000","last_became_peered":"2024-01-16T13:52:59.565543+0000","last_unstale":"2024-01-16T13:53:00.792792+0000","last_undegraded":"2024-01-16T13:53:00.792792+0000","last_fullsized":"2024-01-16T13:53:00.792792+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:25.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562806+0000","last_change":"2024-01-16T13:52:59.562924+0000","last_active":"2024-01-16T13:53:00.562806+0000","last_peered":"2024-01-16T13:53:00.562806+0000","last_clean":"2024-01-16T13:53:00.562806+0000","last_became_active":"2024-01-16T13:52:59.562629+0000","last_became_peered":"2024-01-16T13:52:59.562629+0000","last_unstale":"2024-01-16T13:53:00.562806+0000","last_undegraded":"2024-01-16T13:53:00.562806+0000","last_fullsized":"2024-01-16T13:53:00.562806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:05.512395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.572152+0000","last_change":"2024-01-16T13:52:58.559157+0000","last_active":"2024-01-16T13:52:59.572152+0000","last_peered":"2024-01-16T13:52:59.572152+0000","last_clean":"2024-01-16T13:52:59.572152+0000","last_became_active":"2024-01-16T13:52:58.558913+0000","last_became_peered":"2024-01-16T13:52:58.558913+0000","last_unstale":"2024-01-16T13:52:59.572152+0000","last_undegraded":"2024-01-16T13:52:59.572152+0000","last_fullsized":"2024-01-16T13:52:59.572152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:03.776676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792522+0000","last_change":"2024-01-16T13:52:59.563094+0000","last_active":"2024-01-16T13:53:00.792522+0000","last_peered":"2024-01-16T13:53:00.792522+0000","last_clean":"2024-01-16T13:53:00.792522+0000","last_became_active":"2024-01-16T13:52:59.562819+0000","last_became_peered":"2024-01-16T13:52:59.562819+0000","last_unstale":"2024-01-16T13:53:00.792522+0000","last_undegraded":"2024-01-16T13:53:00.792522+0000","last_fullsized":"2024-01-16T13:53:00.792522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:12:53.889907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562805+0000","last_change":"2024-01-16T13:52:58.557125+0000","last_active":"2024-01-16T13:53:00.562805+0000","last_peered":"2024-01-16T13:53:00.562805+0000","last_clean":"2024-01-16T13:53:00.562805+0000","last_became_active":"2024-01-16T13:52:58.556817+0000","last_became_peered":"2024-01-16T13:52:58.556817+0000","last_unstale":"2024-01-16T13:53:00.562805+0000","last_undegraded":"2024-01-16T13:53:00.562805+0000","last_fullsized":"2024-01-16T13:53:00.562805+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:54.995534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792954+0000","last_change":"2024-01-16T13:52:59.570308+0000","last_active":"2024-01-16T13:53:00.792954+0000","last_peered":"2024-01-16T13:53:00.792954+0000","last_clean":"2024-01-16T13:53:00.792954+0000","last_became_active":"2024-01-16T13:52:59.570029+0000","last_became_peered":"2024-01-16T13:52:59.570029+0000","last_unstale":"2024-01-16T13:53:00.792954+0000","last_undegraded":"2024-01-16T13:53:00.792954+0000","last_fullsized":"2024-01-16T13:53:00.792954+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:53.015251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.576132+0000","last_change":"2024-01-16T13:52:58.562496+0000","last_active":"2024-01-16T13:52:59.576132+0000","last_peered":"2024-01-16T13:52:59.576132+0000","last_clean":"2024-01-16T13:52:59.576132+0000","last_became_active":"2024-01-16T13:52:58.562231+0000","last_became_peered":"2024-01-16T13:52:58.562231+0000","last_unstale":"2024-01-16T13:52:59.576132+0000","last_undegraded":"2024-01-16T13:52:59.576132+0000","last_fullsized":"2024-01-16T13:52:59.576132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:16:59.587520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560532+0000","last_change":"2024-01-16T13:52:59.566968+0000","last_active":"2024-01-16T13:53:00.560532+0000","last_peered":"2024-01-16T13:53:00.560532+0000","last_clean":"2024-01-16T13:53:00.560532+0000","last_became_active":"2024-01-16T13:52:59.566268+0000","last_became_peered":"2024-01-16T13:52:59.566268+0000","last_unstale":"2024-01-16T13:53:00.560532+0000","last_undegraded":"2024-01-16T13:53:00.560532+0000","last_fullsized":"2024-01-16T13:53:00.560532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:02.720813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.572245+0000","last_change":"2024-01-16T13:52:58.562906+0000","last_active":"2024-01-16T13:52:59.572245+0000","last_peered":"2024-01-16T13:52:59.572245+0000","last_clean":"2024-01-16T13:52:59.572245+0000","last_became_active":"2024-01-16T13:52:58.562517+0000","last_became_peered":"2024-01-16T13:52:58.562517+0000","last_unstale":"2024-01-16T13:52:59.572245+0000","last_undegraded":"2024-01-16T13:52:59.572245+0000","last_fullsized":"2024-01-16T13:52:59.572245+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:27.869390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792327+0000","last_change":"2024-01-16T13:52:59.564090+0000","last_active":"2024-01-16T13:53:00.792327+0000","last_peered":"2024-01-16T13:53:00.792327+0000","last_clean":"2024-01-16T13:53:00.792327+0000","last_became_active":"2024-01-16T13:52:59.563810+0000","last_became_peered":"2024-01-16T13:52:59.563810+0000","last_unstale":"2024-01-16T13:53:00.792327+0000","last_undegraded":"2024-01-16T13:53:00.792327+0000","last_fullsized":"2024-01-16T13:53:00.792327+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:56.032353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.576352+0000","last_change":"2024-01-16T13:52:58.566758+0000","last_active":"2024-01-16T13:52:59.576352+0000","last_peered":"2024-01-16T13:52:59.576352+0000","last_clean":"2024-01-16T13:52:59.576352+0000","last_became_active":"2024-01-16T13:52:58.566541+0000","last_became_peered":"2024-01-16T13:52:58.566541+0000","last_unstale":"2024-01-16T13:52:59.576352+0000","last_undegraded":"2024-01-16T13:52:59.576352+0000","last_fullsized":"2024-01-16T13:52:59.576352+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:32.906444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563140+0000","last_change":"2024-01-16T13:52:59.566605+0000","last_active":"2024-01-16T13:53:00.563140+0000","last_peered":"2024-01-16T13:53:00.563140+0000","last_clean":"2024-01-16T13:53:00.563140+0000","last_became_active":"2024-01-16T13:52:59.566324+0000","last_became_peered":"2024-01-16T13:52:59.566324+0000","last_unstale":"2024-01-16T13:53:00.563140+0000","last_undegraded":"2024-01-16T13:53:00.563140+0000","last_fullsized":"2024-01-16T13:53:00.563140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:21.972684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.574564+0000","last_change":"2024-01-16T13:52:58.561808+0000","last_active":"2024-01-16T13:52:59.574564+0000","last_peered":"2024-01-16T13:52:59.574564+0000","last_clean":"2024-01-16T13:52:59.574564+0000","last_became_active":"2024-01-16T13:52:58.561490+0000","last_became_peered":"2024-01-16T13:52:58.561490+0000","last_unstale":"2024-01-16T13:52:59.574564+0000","last_undegraded":"2024-01-16T13:52:59.574564+0000","last_fullsized":"2024-01-16T13:52:59.574564+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:23.472571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792031+0000","last_change":"2024-01-16T13:52:59.559248+0000","last_active":"2024-01-16T13:53:00.792031+0000","last_peered":"2024-01-16T13:53:00.792031+0000","last_clean":"2024-01-16T13:53:00.792031+0000","last_became_active":"2024-01-16T13:52:59.558961+0000","last_became_peered":"2024-01-16T13:52:59.558961+0000","last_unstale":"2024-01-16T13:53:00.792031+0000","last_undegraded":"2024-01-16T13:53:00.792031+0000","last_fullsized":"2024-01-16T13:53:00.792031+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:50:00.627242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547589+0000","last_change":"2024-01-16T13:52:58.561453+0000","last_active":"2024-01-16T13:52:59.547589+0000","last_peered":"2024-01-16T13:52:59.547589+0000","last_clean":"2024-01-16T13:52:59.547589+0000","last_became_active":"2024-01-16T13:52:58.560956+0000","last_became_peered":"2024-01-16T13:52:58.560956+0000","last_unstale":"2024-01-16T13:52:59.547589+0000","last_undegraded":"2024-01-16T13:52:59.547589+0000","last_fullsized":"2024-01-16T13:52:59.547589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:03.019234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560198+0000","last_change":"2024-01-16T13:52:59.565380+0000","last_active":"2024-01-16T13:53:00.560198+0000","last_peered":"2024-01-16T13:53:00.560198+0000","last_clean":"2024-01-16T13:53:00.560198+0000","last_became_active":"2024-01-16T13:52:59.564986+0000","last_became_peered":"2024-01-16T13:52:59.564986+0000","last_unstale":"2024-01-16T13:53:00.560198+0000","last_undegraded":"2024-01-16T13:53:00.560198+0000","last_fullsized":"2024-01-16T13:53:00.560198+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:57.472694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.546031+0000","last_change":"2024-01-16T13:52:58.562274+0000","last_active":"2024-01-16T13:52:59.546031+0000","last_peered":"2024-01-16T13:52:59.546031+0000","last_clean":"2024-01-16T13:52:59.546031+0000","last_became_active":"2024-01-16T13:52:58.561978+0000","last_became_peered":"2024-01-16T13:52:58.561978+0000","last_unstale":"2024-01-16T13:52:59.546031+0000","last_undegraded":"2024-01-16T13:52:59.546031+0000","last_fullsized":"2024-01-16T13:52:59.546031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:36:13.374766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563291+0000","last_change":"2024-01-16T13:52:59.557554+0000","last_active":"2024-01-16T13:53:00.563291+0000","last_peered":"2024-01-16T13:53:00.563291+0000","last_clean":"2024-01-16T13:53:00.563291+0000","last_became_active":"2024-01-16T13:52:59.557271+0000","last_became_peered":"2024-01-16T13:52:59.557271+0000","last_unstale":"2024-01-16T13:53:00.563291+0000","last_undegraded":"2024-01-16T13:53:00.563291+0000","last_fullsized":"2024-01-16T13:53:00.563291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:14.286572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.548340+0000","last_change":"2024-01-16T13:52:58.566147+0000","last_active":"2024-01-16T13:52:59.548340+0000","last_peered":"2024-01-16T13:52:59.548340+0000","last_clean":"2024-01-16T13:52:59.548340+0000","last_became_active":"2024-01-16T13:52:58.565861+0000","last_became_peered":"2024-01-16T13:52:58.565861+0000","last_unstale":"2024-01-16T13:52:59.548340+0000","last_undegraded":"2024-01-16T13:52:59.548340+0000","last_fullsized":"2024-01-16T13:52:59.548340+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:19.323195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609904+0000","last_change":"2024-01-16T13:52:59.569867+0000","last_active":"2024-01-16T13:53:00.609904+0000","last_peered":"2024-01-16T13:53:00.609904+0000","last_clean":"2024-01-16T13:53:00.609904+0000","last_became_active":"2024-01-16T13:52:59.569611+0000","last_became_peered":"2024-01-16T13:52:59.569611+0000","last_unstale":"2024-01-16T13:53:00.609904+0000","last_undegraded":"2024-01-16T13:53:00.609904+0000","last_fullsized":"2024-01-16T13:53:00.609904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:30.894670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562879+0000","last_change":"2024-01-16T13:52:58.562964+0000","last_active":"2024-01-16T13:53:00.562879+0000","last_peered":"2024-01-16T13:53:00.562879+0000","last_clean":"2024-01-16T13:53:00.562879+0000","last_became_active":"2024-01-16T13:52:58.562709+0000","last_became_peered":"2024-01-16T13:52:58.562709+0000","last_unstale":"2024-01-16T13:53:00.562879+0000","last_undegraded":"2024-01-16T13:53:00.562879+0000","last_fullsized":"2024-01-16T13:53:00.562879+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:38.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609659+0000","last_change":"2024-01-16T13:52:59.568844+0000","last_active":"2024-01-16T13:53:00.609659+0000","last_peered":"2024-01-16T13:53:00.609659+0000","last_clean":"2024-01-16T13:53:00.609659+0000","last_became_active":"2024-01-16T13:52:59.568587+0000","last_became_peered":"2024-01-16T13:52:59.568587+0000","last_unstale":"2024-01-16T13:53:00.609659+0000","last_undegraded":"2024-01-16T13:53:00.609659+0000","last_fullsized":"2024-01-16T13:53:00.609659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:25.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.563899+0000","last_change":"2024-01-16T13:52:58.556439+0000","last_active":"2024-01-16T13:52:59.563899+0000","last_peered":"2024-01-16T13:52:59.563899+0000","last_clean":"2024-01-16T13:52:59.563899+0000","last_became_active":"2024-01-16T13:52:58.556198+0000","last_became_peered":"2024-01-16T13:52:58.556198+0000","last_unstale":"2024-01-16T13:52:59.563899+0000","last_undegraded":"2024-01-16T13:52:59.563899+0000","last_fullsized":"2024-01-16T13:52:59.563899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:01.718215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563331+0000","last_change":"2024-01-16T13:52:59.570037+0000","last_active":"2024-01-16T13:53:00.563331+0000","last_peered":"2024-01-16T13:53:00.563331+0000","last_clean":"2024-01-16T13:53:00.563331+0000","last_became_active":"2024-01-16T13:52:59.569239+0000","last_became_peered":"2024-01-16T13:52:59.569239+0000","last_unstale":"2024-01-16T13:53:00.563331+0000","last_undegraded":"2024-01-16T13:53:00.563331+0000","last_fullsized":"2024-01-16T13:53:00.563331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:08.471154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.578050+0000","last_change":"2024-01-16T13:52:58.563418+0000","last_active":"2024-01-16T13:52:59.578050+0000","last_peered":"2024-01-16T13:52:59.578050+0000","last_clean":"2024-01-16T13:52:59.578050+0000","last_became_active":"2024-01-16T13:52:58.563149+0000","last_became_peered":"2024-01-16T13:52:58.563149+0000","last_unstale":"2024-01-16T13:52:59.578050+0000","last_undegraded":"2024-01-16T13:52:59.578050+0000","last_fullsized":"2024-01-16T13:52:59.578050+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:56.470097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.559947+0000","last_change":"2024-01-16T13:52:59.563254+0000","last_active":"2024-01-16T13:53:00.559947+0000","last_peered":"2024-01-16T13:53:00.559947+0000","last_clean":"2024-01-16T13:53:00.559947+0000","last_became_active":"2024-01-16T13:52:59.563001+0000","last_became_peered":"2024-01-16T13:52:59.563001+0000","last_unstale":"2024-01-16T13:53:00.559947+0000","last_undegraded":"2024-01-16T13:53:00.559947+0000","last_fullsized":"2024-01-16T13:53:00.559947+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:20.325793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.573503+0000","last_change":"2024-01-16T13:52:58.562574+0000","last_active":"2024-01-16T13:52:59.573503+0000","last_peered":"2024-01-16T13:52:59.573503+0000","last_clean":"2024-01-16T13:52:59.573503+0000","last_became_active":"2024-01-16T13:52:58.562303+0000","last_became_peered":"2024-01-16T13:52:58.562303+0000","last_unstale":"2024-01-16T13:52:59.573503+0000","last_undegraded":"2024-01-16T13:52:59.573503+0000","last_fullsized":"2024-01-16T13:52:59.573503+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:29.876112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609735+0000","last_change":"2024-01-16T13:52:59.566100+0000","last_active":"2024-01-16T13:53:00.609735+0000","last_peered":"2024-01-16T13:53:00.609735+0000","last_clean":"2024-01-16T13:53:00.609735+0000","last_became_active":"2024-01-16T13:52:59.565846+0000","last_became_peered":"2024-01-16T13:52:59.565846+0000","last_unstale":"2024-01-16T13:53:00.609735+0000","last_undegraded":"2024-01-16T13:53:00.609735+0000","last_fullsized":"2024-01-16T13:53:00.609735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:14.286572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.554048+0000","last_change":"2024-01-16T13:52:58.559322+0000","last_active":"2024-01-16T13:52:59.554048+0000","last_peered":"2024-01-16T13:52:59.554048+0000","last_clean":"2024-01-16T13:52:59.554048+0000","last_became_active":"2024-01-16T13:52:58.558802+0000","last_became_peered":"2024-01-16T13:52:58.558802+0000","last_unstale":"2024-01-16T13:52:59.554048+0000","last_undegraded":"2024-01-16T13:52:59.554048+0000","last_fullsized":"2024-01-16T13:52:59.554048+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:38.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792614+0000","last_change":"2024-01-16T13:52:59.564532+0000","last_active":"2024-01-16T13:53:00.792614+0000","last_peered":"2024-01-16T13:53:00.792614+0000","last_clean":"2024-01-16T13:53:00.792614+0000","last_became_active":"2024-01-16T13:52:59.564247+0000","last_became_peered":"2024-01-16T13:52:59.564247+0000","last_unstale":"2024-01-16T13:53:00.792614+0000","last_undegraded":"2024-01-16T13:53:00.792614+0000","last_fullsized":"2024-01-16T13:53:00.792614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:21.972684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547723+0000","last_change":"2024-01-16T13:52:58.561629+0000","last_active":"2024-01-16T13:52:59.547723+0000","last_peered":"2024-01-16T13:52:59.547723+0000","last_clean":"2024-01-16T13:52:59.547723+0000","last_became_active":"2024-01-16T13:52:58.561193+0000","last_became_peered":"2024-01-16T13:52:58.561193+0000","last_unstale":"2024-01-16T13:52:59.547723+0000","last_undegraded":"2024-01-16T13:52:59.547723+0000","last_fullsized":"2024-01-16T13:52:59.547723+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:49:59.624644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792417+0000","last_change":"2024-01-16T13:52:59.564246+0000","last_active":"2024-01-16T13:53:00.792417+0000","last_peered":"2024-01-16T13:53:00.792417+0000","last_clean":"2024-01-16T13:53:00.792417+0000","last_became_active":"2024-01-16T13:52:59.563968+0000","last_became_peered":"2024-01-16T13:52:59.563968+0000","last_unstale":"2024-01-16T13:53:00.792417+0000","last_undegraded":"2024-01-16T13:53:00.792417+0000","last_fullsized":"2024-01-16T13:53:00.792417+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:38.154103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.577048+0000","last_change":"2024-01-16T13:52:58.566564+0000","last_active":"2024-01-16T13:52:59.577048+0000","last_peered":"2024-01-16T13:52:59.577048+0000","last_clean":"2024-01-16T13:52:59.577048+0000","last_became_active":"2024-01-16T13:52:58.566241+0000","last_became_peered":"2024-01-16T13:52:58.566241+0000","last_unstale":"2024-01-16T13:52:59.577048+0000","last_undegraded":"2024-01-16T13:52:59.577048+0000","last_fullsized":"2024-01-16T13:52:59.577048+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:13.685725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793086+0000","last_change":"2024-01-16T13:52:59.557019+0000","last_active":"2024-01-16T13:53:00.793086+0000","last_peered":"2024-01-16T13:53:00.793086+0000","last_clean":"2024-01-16T13:53:00.793086+0000","last_became_active":"2024-01-16T13:52:59.556754+0000","last_became_peered":"2024-01-16T13:52:59.556754+0000","last_unstale":"2024-01-16T13:53:00.793086+0000","last_undegraded":"2024-01-16T13:53:00.793086+0000","last_fullsized":"2024-01-16T13:53:00.793086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:57.472694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.576731+0000","last_change":"2024-01-16T13:52:58.561600+0000","last_active":"2024-01-16T13:52:59.576731+0000","last_peered":"2024-01-16T13:52:59.576731+0000","last_clean":"2024-01-16T13:52:59.576731+0000","last_became_active":"2024-01-16T13:52:58.561333+0000","last_became_peered":"2024-01-16T13:52:58.561333+0000","last_unstale":"2024-01-16T13:52:59.576731+0000","last_undegraded":"2024-01-16T13:52:59.576731+0000","last_fullsized":"2024-01-16T13:52:59.576731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:04.509797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563723+0000","last_change":"2024-01-16T13:52:59.554017+0000","last_active":"2024-01-16T13:53:00.563723+0000","last_peered":"2024-01-16T13:53:00.563723+0000","last_clean":"2024-01-16T13:53:00.563723+0000","last_became_active":"2024-01-16T13:52:59.553676+0000","last_became_peered":"2024-01-16T13:52:59.553676+0000","last_unstale":"2024-01-16T13:53:00.563723+0000","last_undegraded":"2024-01-16T13:53:00.563723+0000","last_fullsized":"2024-01-16T13:53:00.563723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:34.382768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.568454+0000","last_change":"2024-01-16T13:52:58.562411+0000","last_active":"2024-01-16T13:52:59.568454+0000","last_peered":"2024-01-16T13:52:59.568454+0000","last_clean":"2024-01-16T13:52:59.568454+0000","last_became_active":"2024-01-16T13:52:58.562031+0000","last_became_peered":"2024-01-16T13:52:58.562031+0000","last_unstale":"2024-01-16T13:52:59.568454+0000","last_undegraded":"2024-01-16T13:52:59.568454+0000","last_fullsized":"2024-01-16T13:52:59.568454+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:14.140338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560237+0000","last_change":"2024-01-16T13:52:59.564287+0000","last_active":"2024-01-16T13:53:00.560237+0000","last_peered":"2024-01-16T13:53:00.560237+0000","last_clean":"2024-01-16T13:53:00.560237+0000","last_became_active":"2024-01-16T13:52:59.564029+0000","last_became_peered":"2024-01-16T13:52:59.564029+0000","last_unstale":"2024-01-16T13:53:00.560237+0000","last_undegraded":"2024-01-16T13:53:00.560237+0000","last_fullsized":"2024-01-16T13:53:00.560237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:00.590118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.570508+0000","last_change":"2024-01-16T13:52:58.562668+0000","last_active":"2024-01-16T13:52:59.570508+0000","last_peered":"2024-01-16T13:52:59.570508+0000","last_clean":"2024-01-16T13:52:59.570508+0000","last_became_active":"2024-01-16T13:52:58.562432+0000","last_became_peered":"2024-01-16T13:52:58.562432+0000","last_unstale":"2024-01-16T13:52:59.570508+0000","last_undegraded":"2024-01-16T13:52:59.570508+0000","last_fullsized":"2024-01-16T13:52:59.570508+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:04.946684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791140+0000","last_change":"2024-01-16T13:52:59.566650+0000","last_active":"2024-01-16T13:53:00.791140+0000","last_peered":"2024-01-16T13:53:00.791140+0000","last_clean":"2024-01-16T13:53:00.791140+0000","last_became_active":"2024-01-16T13:52:59.566395+0000","last_became_peered":"2024-01-16T13:52:59.566395+0000","last_unstale":"2024-01-16T13:53:00.791140+0000","last_undegraded":"2024-01-16T13:53:00.791140+0000","last_fullsized":"2024-01-16T13:53:00.791140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:25.689016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.546600+0000","last_change":"2024-01-16T13:52:58.566549+0000","last_active":"2024-01-16T13:52:59.546600+0000","last_peered":"2024-01-16T13:52:59.546600+0000","last_clean":"2024-01-16T13:52:59.546600+0000","last_became_active":"2024-01-16T13:52:58.566269+0000","last_became_peered":"2024-01-16T13:52:58.566269+0000","last_unstale":"2024-01-16T13:52:59.546600+0000","last_undegraded":"2024-01-16T13:52:59.546600+0000","last_fullsized":"2024-01-16T13:52:59.546600+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:49.846088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791078+0000","last_change":"2024-01-16T13:52:59.566738+0000","last_active":"2024-01-16T13:53:00.791078+0000","last_peered":"2024-01-16T13:53:00.791078+0000","last_clean":"2024-01-16T13:53:00.791078+0000","last_became_active":"2024-01-16T13:52:59.566462+0000","last_became_peered":"2024-01-16T13:52:59.566462+0000","last_unstale":"2024-01-16T13:53:00.791078+0000","last_undegraded":"2024-01-16T13:53:00.791078+0000","last_fullsized":"2024-01-16T13:53:00.791078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:36:14.377364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562804+0000","last_change":"2024-01-16T13:52:58.559782+0000","last_active":"2024-01-16T13:53:00.562804+0000","last_peered":"2024-01-16T13:53:00.562804+0000","last_clean":"2024-01-16T13:53:00.562804+0000","last_became_active":"2024-01-16T13:52:58.559472+0000","last_became_peered":"2024-01-16T13:52:58.559472+0000","last_unstale":"2024-01-16T13:53:00.562804+0000","last_undegraded":"2024-01-16T13:53:00.562804+0000","last_fullsized":"2024-01-16T13:53:00.562804+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:37.151505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791132+0000","last_change":"2024-01-16T13:52:59.569265+0000","last_active":"2024-01-16T13:53:00.791132+0000","last_peered":"2024-01-16T13:53:00.791132+0000","last_clean":"2024-01-16T13:53:00.791132+0000","last_became_active":"2024-01-16T13:52:59.569074+0000","last_became_peered":"2024-01-16T13:52:59.569074+0000","last_unstale":"2024-01-16T13:53:00.791132+0000","last_undegraded":"2024-01-16T13:53:00.791132+0000","last_fullsized":"2024-01-16T13:53:00.791132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:05.949282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.553273+0000","last_change":"2024-01-16T13:52:58.563508+0000","last_active":"2024-01-16T13:52:59.553273+0000","last_peered":"2024-01-16T13:52:59.553273+0000","last_clean":"2024-01-16T13:52:59.553273+0000","last_became_active":"2024-01-16T13:52:58.562973+0000","last_became_peered":"2024-01-16T13:52:58.562973+0000","last_unstale":"2024-01-16T13:52:59.553273+0000","last_undegraded":"2024-01-16T13:52:59.553273+0000","last_fullsized":"2024-01-16T13:52:59.553273+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:38.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791082+0000","last_change":"2024-01-16T13:52:59.569332+0000","last_active":"2024-01-16T13:53:00.791082+0000","last_peered":"2024-01-16T13:53:00.791082+0000","last_clean":"2024-01-16T13:53:00.791082+0000","last_became_active":"2024-01-16T13:52:59.569036+0000","last_became_peered":"2024-01-16T13:52:59.569036+0000","last_unstale":"2024-01-16T13:53:00.791082+0000","last_undegraded":"2024-01-16T13:53:00.791082+0000","last_fullsized":"2024-01-16T13:53:00.791082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:24.475169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.546853+0000","last_change":"2024-01-16T13:52:58.563616+0000","last_active":"2024-01-16T13:52:59.546853+0000","last_peered":"2024-01-16T13:52:59.546853+0000","last_clean":"2024-01-16T13:52:59.546853+0000","last_became_active":"2024-01-16T13:52:58.563302+0000","last_became_peered":"2024-01-16T13:52:58.563302+0000","last_unstale":"2024-01-16T13:52:59.546853+0000","last_undegraded":"2024-01-16T13:52:59.546853+0000","last_fullsized":"2024-01-16T13:52:59.546853+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:24.686418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792793+0000","last_change":"2024-01-16T13:52:59.567226+0000","last_active":"2024-01-16T13:53:00.792793+0000","last_peered":"2024-01-16T13:53:00.792793+0000","last_clean":"2024-01-16T13:53:00.792793+0000","last_became_active":"2024-01-16T13:52:59.566659+0000","last_became_peered":"2024-01-16T13:52:59.566659+0000","last_unstale":"2024-01-16T13:53:00.792793+0000","last_undegraded":"2024-01-16T13:53:00.792793+0000","last_fullsized":"2024-01-16T13:53:00.792793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:14.286572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.548876+0000","last_change":"2024-01-16T13:52:58.559155+0000","last_active":"2024-01-16T13:52:59.548876+0000","last_peered":"2024-01-16T13:52:59.548876+0000","last_clean":"2024-01-16T13:52:59.548876+0000","last_became_active":"2024-01-16T13:52:58.558863+0000","last_became_peered":"2024-01-16T13:52:58.558863+0000","last_unstale":"2024-01-16T13:52:59.548876+0000","last_undegraded":"2024-01-16T13:52:59.548876+0000","last_fullsized":"2024-01-16T13:52:59.548876+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:56.470097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791138+0000","last_change":"2024-01-16T13:52:59.567708+0000","last_active":"2024-01-16T13:53:00.791138+0000","last_peered":"2024-01-16T13:53:00.791138+0000","last_clean":"2024-01-16T13:53:00.791138+0000","last_became_active":"2024-01-16T13:52:59.567461+0000","last_became_peered":"2024-01-16T13:52:59.567461+0000","last_unstale":"2024-01-16T13:53:00.791138+0000","last_undegraded":"2024-01-16T13:53:00.791138+0000","last_fullsized":"2024-01-16T13:53:00.791138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:50.848686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.572479+0000","last_change":"2024-01-16T13:52:58.559075+0000","last_active":"2024-01-16T13:52:59.572479+0000","last_peered":"2024-01-16T13:52:59.572479+0000","last_clean":"2024-01-16T13:52:59.572479+0000","last_became_active":"2024-01-16T13:52:58.558722+0000","last_became_peered":"2024-01-16T13:52:58.558722+0000","last_unstale":"2024-01-16T13:52:59.572479+0000","last_undegraded":"2024-01-16T13:52:59.572479+0000","last_fullsized":"2024-01-16T13:52:59.572479+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:13.685725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563669+0000","last_change":"2024-01-16T13:52:59.569889+0000","last_active":"2024-01-16T13:53:00.563669+0000","last_peered":"2024-01-16T13:53:00.563669+0000","last_clean":"2024-01-16T13:53:00.563669+0000","last_became_active":"2024-01-16T13:52:59.569282+0000","last_became_peered":"2024-01-16T13:52:59.569282+0000","last_unstale":"2024-01-16T13:53:00.563669+0000","last_undegraded":"2024-01-16T13:53:00.563669+0000","last_fullsized":"2024-01-16T13:53:00.563669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:30.894670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:01.101751+0000","last_change":"2024-01-16T13:52:58.559563+0000","last_active":"2024-01-16T13:53:01.101751+0000","last_peered":"2024-01-16T13:53:01.101751+0000","last_clean":"2024-01-16T13:53:01.101751+0000","last_became_active":"2024-01-16T13:52:58.559233+0000","last_became_peered":"2024-01-16T13:52:58.559233+0000","last_unstale":"2024-01-16T13:53:01.101751+0000","last_undegraded":"2024-01-16T13:53:01.101751+0000","last_fullsized":"2024-01-16T13:53:01.101751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:03.776676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562783+0000","last_change":"2024-01-16T13:52:59.572336+0000","last_active":"2024-01-16T13:53:00.562783+0000","last_peered":"2024-01-16T13:53:00.562783+0000","last_clean":"2024-01-16T13:53:00.562783+0000","last_became_active":"2024-01-16T13:52:59.572074+0000","last_became_peered":"2024-01-16T13:52:59.572074+0000","last_unstale":"2024-01-16T13:53:00.562783+0000","last_undegraded":"2024-01-16T13:53:00.562783+0000","last_fullsized":"2024-01-16T13:53:00.562783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:14.688323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.554523+0000","last_change":"2024-01-16T13:52:58.558949+0000","last_active":"2024-01-16T13:52:59.554523+0000","last_peered":"2024-01-16T13:52:59.554523+0000","last_clean":"2024-01-16T13:52:59.554523+0000","last_became_active":"2024-01-16T13:52:58.558648+0000","last_became_peered":"2024-01-16T13:52:58.558648+0000","last_unstale":"2024-01-16T13:52:59.554523+0000","last_undegraded":"2024-01-16T13:52:59.554523+0000","last_fullsized":"2024-01-16T13:52:59.554523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:32.906444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792034+0000","last_change":"2024-01-16T13:52:59.566492+0000","last_active":"2024-01-16T13:53:00.792034+0000","last_peered":"2024-01-16T13:53:00.792034+0000","last_clean":"2024-01-16T13:53:00.792034+0000","last_became_active":"2024-01-16T13:52:59.566221+0000","last_became_peered":"2024-01-16T13:52:59.566221+0000","last_unstale":"2024-01-16T13:53:00.792034+0000","last_undegraded":"2024-01-16T13:53:00.792034+0000","last_fullsized":"2024-01-16T13:53:00.792034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:04.021832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562737+0000","last_change":"2024-01-16T13:52:58.553716+0000","last_active":"2024-01-16T13:53:00.562737+0000","last_peered":"2024-01-16T13:53:00.562737+0000","last_clean":"2024-01-16T13:53:00.562737+0000","last_became_active":"2024-01-16T13:52:58.553078+0000","last_became_peered":"2024-01-16T13:52:58.553078+0000","last_unstale":"2024-01-16T13:53:00.562737+0000","last_undegraded":"2024-01-16T13:53:00.562737+0000","last_fullsized":"2024-01-16T13:53:00.562737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:29.876112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562911+0000","last_change":"2024-01-16T13:52:59.566180+0000","last_active":"2024-01-16T13:53:00.562911+0000","last_peered":"2024-01-16T13:53:00.562911+0000","last_clean":"2024-01-16T13:53:00.562911+0000","last_became_active":"2024-01-16T13:52:59.565847+0000","last_became_peered":"2024-01-16T13:52:59.565847+0000","last_unstale":"2024-01-16T13:53:00.562911+0000","last_undegraded":"2024-01-16T13:53:00.562911+0000","last_fullsized":"2024-01-16T13:53:00.562911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:27.471640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.555191+0000","last_change":"2024-01-16T13:52:58.559484+0000","last_active":"2024-01-16T13:52:59.555191+0000","last_peered":"2024-01-16T13:52:59.555191+0000","last_clean":"2024-01-16T13:52:59.555191+0000","last_became_active":"2024-01-16T13:52:58.559084+0000","last_became_peered":"2024-01-16T13:52:58.559084+0000","last_unstale":"2024-01-16T13:52:59.555191+0000","last_undegraded":"2024-01-16T13:52:59.555191+0000","last_fullsized":"2024-01-16T13:52:59.555191+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:12:52.887309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609403+0000","last_change":"2024-01-16T13:52:59.564927+0000","last_active":"2024-01-16T13:53:00.609403+0000","last_peered":"2024-01-16T13:53:00.609403+0000","last_clean":"2024-01-16T13:53:00.609403+0000","last_became_active":"2024-01-16T13:52:59.564679+0000","last_became_peered":"2024-01-16T13:52:59.564679+0000","last_unstale":"2024-01-16T13:53:00.609403+0000","last_undegraded":"2024-01-16T13:53:00.609403+0000","last_fullsized":"2024-01-16T13:53:00.609403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:18:18.563379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547015+0000","last_change":"2024-01-16T13:52:58.559066+0000","last_active":"2024-01-16T13:52:59.547015+0000","last_peered":"2024-01-16T13:52:59.547015+0000","last_clean":"2024-01-16T13:52:59.547015+0000","last_became_active":"2024-01-16T13:52:58.558776+0000","last_became_peered":"2024-01-16T13:52:58.558776+0000","last_unstale":"2024-01-16T13:52:59.547015+0000","last_undegraded":"2024-01-16T13:52:59.547015+0000","last_fullsized":"2024-01-16T13:52:59.547015+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:54.995534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792349+0000","last_change":"2024-01-16T13:52:59.553174+0000","last_active":"2024-01-16T13:53:00.792349+0000","last_peered":"2024-01-16T13:53:00.792349+0000","last_clean":"2024-01-16T13:53:00.792349+0000","last_became_active":"2024-01-16T13:52:59.552849+0000","last_became_peered":"2024-01-16T13:52:59.552849+0000","last_unstale":"2024-01-16T13:53:00.792349+0000","last_undegraded":"2024-01-16T13:53:00.792349+0000","last_fullsized":"2024-01-16T13:53:00.792349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:05.512395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547961+0000","last_change":"2024-01-16T13:52:58.561252+0000","last_active":"2024-01-16T13:52:59.547961+0000","last_peered":"2024-01-16T13:52:59.547961+0000","last_clean":"2024-01-16T13:52:59.547961+0000","last_became_active":"2024-01-16T13:52:58.560969+0000","last_became_peered":"2024-01-16T13:52:58.560969+0000","last_unstale":"2024-01-16T13:52:59.547961+0000","last_undegraded":"2024-01-16T13:52:59.547961+0000","last_fullsized":"2024-01-16T13:52:59.547961+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:04.509797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792085+0000","last_change":"2024-01-16T13:52:59.569488+0000","last_active":"2024-01-16T13:53:00.792085+0000","last_peered":"2024-01-16T13:53:00.792085+0000","last_clean":"2024-01-16T13:53:00.792085+0000","last_became_active":"2024-01-16T13:52:59.569033+0000","last_became_peered":"2024-01-16T13:52:59.569033+0000","last_unstale":"2024-01-16T13:53:00.792085+0000","last_undegraded":"2024-01-16T13:53:00.792085+0000","last_fullsized":"2024-01-16T13:53:00.792085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:53.788479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.552892+0000","last_change":"2024-01-16T13:52:58.562363+0000","last_active":"2024-01-16T13:52:59.552892+0000","last_peered":"2024-01-16T13:52:59.552892+0000","last_clean":"2024-01-16T13:52:59.552892+0000","last_became_active":"2024-01-16T13:52:58.562073+0000","last_became_peered":"2024-01-16T13:52:58.562073+0000","last_unstale":"2024-01-16T13:52:59.552892+0000","last_undegraded":"2024-01-16T13:52:59.552892+0000","last_fullsized":"2024-01-16T13:52:59.552892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:54.995534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":173,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.552274+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:59.552274+0000","last_peered":"2024-01-16T13:52:59.552274+0000","last_clean":"2024-01-16T13:52:59.552274+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:59.552274+0000","last_undegraded":"2024-01-16T13:52:59.552274+0000","last_fullsized":"2024-01-16T13:52:59.552274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":734546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.497}]},{"osd":3,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.73199999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.77600000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.83099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.627}]},{"osd":1,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.60799999999999998}]},{"osd":2,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.505},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48399999999999999}]},{"osd":5,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.44}]},{"osd":7,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":145722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.371},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.374,"5min":0.374,"15min":0.374},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.80400000000000005}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.85499999999999998}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.746}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":734456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.376},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36299999999999999}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.69099999999999995}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.55400000000000005}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.67000000000000004}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.81000000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":144158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.34000000000000002}]},{"osd":1,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.25900000000000001}]},{"osd":3,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.502}]},{"osd":5,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51300000000000001}]},{"osd":6,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46700000000000003}]},{"osd":7,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.437}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":145722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.40400000000000003}]},{"osd":1,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.54900000000000004}]},{"osd":2,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.38100000000000001}]},{"osd":4,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56699999999999995}]},{"osd":6,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45500000000000002}]},{"osd":7,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.44400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.55700000000000005}]},{"osd":1,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":6.8289999999999997,"5min":6.8289999999999997,"15min":6.8289999999999997},"last":0.57899999999999996}]},{"osd":2,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.843,"5min":5.843,"15min":5.843},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":6.3079999999999998,"5min":6.3079999999999998,"15min":6.3079999999999998},"last":0.73799999999999999}]},{"osd":3,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":6.7690000000000001,"5min":6.7690000000000001,"15min":6.7690000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":0.71799999999999997}]},{"osd":5,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.60599999999999998}]},{"osd":6,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.79600000000000004}]},{"osd":7,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.78000000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51900000000000002}]},{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57699999999999996}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.36099999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.38},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46100000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.50700000000000001}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.48299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:53:08.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T13:53:08.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.346+0000 7ffb15158700 1 Processor -- start 2024-01-16T13:53:08.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.346+0000 7ffb15158700 1 -- start start 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb15158700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ffb1015ddd0 0x7ffb100b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb15158700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb1015e1b0 0x7ffb100cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb15158700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100b1d40 con 0x7ffb100d1b30 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb15158700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb100b28c0 con 0x7ffb1015ddd0 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb15158700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1006c000 con 0x7ffb1015e1b0 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0f59e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ffb100d1b30 0x7ffb10161c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:45252/0 (socket says 172.21.15.113:45252) 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0f59e700 1 -- 172.21.15.113:0/2996985399 learned_addr learned my addr 172.21.15.113:0/2996985399 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0ed9d700 1 --2- 172.21.15.113:0/2996985399 >> v2:172.21.15.124:3300/0 conn(0x7ffb1015ddd0 0x7ffb100b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0e59c700 1 --2- 172.21.15.113:0/2996985399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb1015e1b0 0x7ffb100cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0dd9b700 1 -- 172.21.15.113:0/2996985399 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373694815 0 0) 0x7ffb100b1d40 con 0x7ffb100d1b30 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0dd9b700 1 -- 172.21.15.113:0/2996985399 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf80035b0 con 0x7ffb100d1b30 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0e59c700 1 -- 172.21.15.113:0/2996985399 >> v2:172.21.15.124:3300/0 conn(0x7ffb1015ddd0 msgr2=0x7ffb100b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0e59c700 1 --2- 172.21.15.113:0/2996985399 >> v2:172.21.15.124:3300/0 conn(0x7ffb1015ddd0 0x7ffb100b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0e59c700 1 -- 172.21.15.113:0/2996985399 >> v1:172.21.15.113:6789/0 conn(0x7ffb100d1b30 legacy=0x7ffb10161c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0e59c700 1 -- 172.21.15.113:0/2996985399 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb100b1f90 con 0x7ffb1015e1b0 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.347+0000 7ffb0ed9d700 1 --2- 172.21.15.113:0/2996985399 >> v2:172.21.15.124:3300/0 conn(0x7ffb1015ddd0 0x7ffb100b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb0e59c700 1 --2- 172.21.15.113:0/2996985399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb1015e1b0 0x7ffb100cf000 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffb00009840 tx=0x7ffb00009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c85e908e4f5a892 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb0dd9b700 1 -- 172.21.15.113:0/2996985399 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011830 con 0x7ffb1015e1b0 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb0dd9b700 1 -- 172.21.15.113:0/2996985399 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00011990 con 0x7ffb1015e1b0 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb0dd9b700 1 -- 172.21.15.113:0/2996985399 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00011d00 con 0x7ffb1015e1b0 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb15158700 1 -- 172.21.15.113:0/2996985399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb1015e1b0 msgr2=0x7ffb100cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb15158700 1 --2- 172.21.15.113:0/2996985399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb1015e1b0 0x7ffb100cf000 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffb00009840 tx=0x7ffb00009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb15158700 1 -- 172.21.15.113:0/2996985399 shutdown_connections 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb15158700 1 --2- 172.21.15.113:0/2996985399 >> v2:172.21.15.124:3300/0 conn(0x7ffb1015ddd0 0x7ffb100b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb15158700 1 --2- 172.21.15.113:0/2996985399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb1015e1b0 0x7ffb100cf000 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.348+0000 7ffb15158700 1 -- 172.21.15.113:0/2996985399 >> 172.21.15.113:0/2996985399 conn(0x7ffb100731a0 msgr2=0x7ffb1009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.349+0000 7ffb15158700 1 -- 172.21.15.113:0/2996985399 shutdown_connections 2024-01-16T13:53:08.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.349+0000 7ffb15158700 1 -- 172.21.15.113:0/2996985399 wait complete. 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.350+0000 7ffb15158700 1 Processor -- start 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.350+0000 7ffb15158700 1 -- start start 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb15158700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb100d1b30 0x7ffb101587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb15158700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 0x7ffb10158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb15158700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffb1015e1b0 0x7ffb1015f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb15158700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb10159220 con 0x7ffb1015ddd0 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb15158700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101533a0 con 0x7ffb1015e1b0 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb15158700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb10159460 con 0x7ffb100d1b30 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 0x7ffb10158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 0x7ffb10158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50040/0 (socket says 172.21.15.113:50040) 2024-01-16T13:53:08.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 -- 172.21.15.113:0/1284902042 learned_addr learned my addr 172.21.15.113:0/1284902042 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0f59e700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffb1015e1b0 0x7ffb1015f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0ed9d700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb100d1b30 0x7ffb101587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 -- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb100d1b30 msgr2=0x7ffb101587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb100d1b30 0x7ffb101587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 -- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffb1015e1b0 msgr2=0x7ffb1015f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffb1015e1b0 0x7ffb1015f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.351+0000 7ffb0e59c700 1 -- 172.21.15.113:0/1284902042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb040094a0 con 0x7ffb1015ddd0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.352+0000 7ffb0e59c700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 0x7ffb10158ce0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffb00014040 tx=0x7ffb00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.352+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00004550 con 0x7ffb1015ddd0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.352+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0001f070 con 0x7ffb1015ddd0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.352+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb000094a0 con 0x7ffb1015ddd0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.352+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb000038a0 con 0x7ffb1015ddd0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.352+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb100b2ed0 con 0x7ffb1015ddd0 2024-01-16T13:53:08.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.353+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10067f60 con 0x7ffb1015ddd0 2024-01-16T13:53:08.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.353+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffb00003a00 con 0x7ffb1015ddd0 2024-01-16T13:53:08.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.353+0000 7ffaf77fe700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffafc06b350 0x7ffafc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.353+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb00021070 con 0x7ffb1015ddd0 2024-01-16T13:53:08.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.354+0000 7ffb0ed9d700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffafc06b350 0x7ffafc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.354+0000 7ffb0ed9d700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffafc06b350 0x7ffafc06d810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009d70 tx=0x7ffb04009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:08.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.358+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb00056130 con 0x7ffb1015ddd0 2024-01-16T13:53:08.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.542+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb100b2d50 con 0x7ffafc06b350 2024-01-16T13:53:08.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.550+0000 7ffaf77fe700 1 -- 172.21.15.113:0/1284902042 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289439 (secure 0 0 0) 0x7ffb040099c0 con 0x7ffafc06b350 2024-01-16T13:53:08.535 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:08.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffafc06b350 msgr2=0x7ffafc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffafc06b350 0x7ffafc06d810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009d70 tx=0x7ffb04009210 comp rx=0 tx=0).stop 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 msgr2=0x7ffb10158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 0x7ffb10158ce0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffb00014040 tx=0x7ffb00004750 comp rx=0 tx=0).stop 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 shutdown_connections 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffafc06b350 0x7ffafc06d810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffb1015e1b0 0x7ffb1015f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffb100d1b30 0x7ffb101587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 --2- 172.21.15.113:0/1284902042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb1015ddd0 0x7ffb10158ce0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 >> 172.21.15.113:0/1284902042 conn(0x7ffb100731a0 msgr2=0x7ffb1006ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 shutdown_connections 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.553+0000 7ffb15158700 1 -- 172.21.15.113:0/1284902042 wait complete. 2024-01-16T13:53:08.538 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-01-16T13:53:08.547 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-01-16T13:53:08.414370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220324,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754396,"statfs":{"total":773094113280,"available":772868501504,"internally_reserved":0,"allocated":4235264,"data_stored":2338260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.868989"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792374+0000","last_change":"2024-01-16T13:52:59.573148+0000","last_active":"2024-01-16T13:53:00.792374+0000","last_peered":"2024-01-16T13:53:00.792374+0000","last_clean":"2024-01-16T13:53:00.792374+0000","last_became_active":"2024-01-16T13:52:59.568465+0000","last_became_peered":"2024-01-16T13:52:59.568465+0000","last_unstale":"2024-01-16T13:53:00.792374+0000","last_undegraded":"2024-01-16T13:53:00.792374+0000","last_fullsized":"2024-01-16T13:53:00.792374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:30.878710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609874+0000","last_change":"2024-01-16T13:52:59.568178+0000","last_active":"2024-01-16T13:53:00.609874+0000","last_peered":"2024-01-16T13:53:00.609874+0000","last_clean":"2024-01-16T13:53:00.609874+0000","last_became_active":"2024-01-16T13:52:59.567905+0000","last_became_peered":"2024-01-16T13:52:59.567905+0000","last_unstale":"2024-01-16T13:53:00.609874+0000","last_undegraded":"2024-01-16T13:53:00.609874+0000","last_fullsized":"2024-01-16T13:53:00.609874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:34.906681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563165+0000","last_change":"2024-01-16T13:52:59.573614+0000","last_active":"2024-01-16T13:53:00.563165+0000","last_peered":"2024-01-16T13:53:00.563165+0000","last_clean":"2024-01-16T13:53:00.563165+0000","last_became_active":"2024-01-16T13:52:59.573292+0000","last_became_peered":"2024-01-16T13:52:59.573292+0000","last_unstale":"2024-01-16T13:53:00.563165+0000","last_undegraded":"2024-01-16T13:53:00.563165+0000","last_fullsized":"2024-01-16T13:53:00.563165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:12:41.316957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.559981+0000","last_change":"2024-01-16T13:52:59.569475+0000","last_active":"2024-01-16T13:53:00.559981+0000","last_peered":"2024-01-16T13:53:00.559981+0000","last_clean":"2024-01-16T13:53:00.559981+0000","last_became_active":"2024-01-16T13:52:59.569161+0000","last_became_peered":"2024-01-16T13:52:59.569161+0000","last_unstale":"2024-01-16T13:53:00.559981+0000","last_undegraded":"2024-01-16T13:53:00.559981+0000","last_fullsized":"2024-01-16T13:53:00.559981+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:53.015251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791137+0000","last_change":"2024-01-16T13:52:59.564689+0000","last_active":"2024-01-16T13:53:00.791137+0000","last_peered":"2024-01-16T13:53:00.791137+0000","last_clean":"2024-01-16T13:53:00.791137+0000","last_became_active":"2024-01-16T13:52:59.564414+0000","last_became_peered":"2024-01-16T13:52:59.564414+0000","last_unstale":"2024-01-16T13:53:00.791137+0000","last_undegraded":"2024-01-16T13:53:00.791137+0000","last_fullsized":"2024-01-16T13:53:00.791137+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:15.726165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609629+0000","last_change":"2024-01-16T13:52:59.563734+0000","last_active":"2024-01-16T13:53:00.609629+0000","last_peered":"2024-01-16T13:53:00.609629+0000","last_clean":"2024-01-16T13:53:00.609629+0000","last_became_active":"2024-01-16T13:52:59.561315+0000","last_became_peered":"2024-01-16T13:52:59.561315+0000","last_unstale":"2024-01-16T13:53:00.609629+0000","last_undegraded":"2024-01-16T13:53:00.609629+0000","last_fullsized":"2024-01-16T13:53:00.609629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:12:41.316957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792646+0000","last_change":"2024-01-16T13:52:59.567372+0000","last_active":"2024-01-16T13:53:00.792646+0000","last_peered":"2024-01-16T13:53:00.792646+0000","last_clean":"2024-01-16T13:53:00.792646+0000","last_became_active":"2024-01-16T13:52:59.566993+0000","last_became_peered":"2024-01-16T13:52:59.566993+0000","last_unstale":"2024-01-16T13:53:00.792646+0000","last_undegraded":"2024-01-16T13:53:00.792646+0000","last_fullsized":"2024-01-16T13:53:00.792646+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:12:41.316957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792382+0000","last_change":"2024-01-16T13:52:59.566131+0000","last_active":"2024-01-16T13:53:00.792382+0000","last_peered":"2024-01-16T13:53:00.792382+0000","last_clean":"2024-01-16T13:53:00.792382+0000","last_became_active":"2024-01-16T13:52:59.565866+0000","last_became_peered":"2024-01-16T13:52:59.565866+0000","last_unstale":"2024-01-16T13:53:00.792382+0000","last_undegraded":"2024-01-16T13:53:00.792382+0000","last_fullsized":"2024-01-16T13:53:00.792382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:32:24.125629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792912+0000","last_change":"2024-01-16T13:52:59.570127+0000","last_active":"2024-01-16T13:53:00.792912+0000","last_peered":"2024-01-16T13:53:00.792912+0000","last_clean":"2024-01-16T13:53:00.792912+0000","last_became_active":"2024-01-16T13:52:59.569670+0000","last_became_peered":"2024-01-16T13:52:59.569670+0000","last_unstale":"2024-01-16T13:53:00.792912+0000","last_undegraded":"2024-01-16T13:53:00.792912+0000","last_fullsized":"2024-01-16T13:53:00.792912+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:20.325793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560200+0000","last_change":"2024-01-16T13:52:59.566004+0000","last_active":"2024-01-16T13:53:00.560200+0000","last_peered":"2024-01-16T13:53:00.560200+0000","last_clean":"2024-01-16T13:53:00.560200+0000","last_became_active":"2024-01-16T13:52:59.565695+0000","last_became_peered":"2024-01-16T13:52:59.565695+0000","last_unstale":"2024-01-16T13:53:00.560200+0000","last_undegraded":"2024-01-16T13:53:00.560200+0000","last_fullsized":"2024-01-16T13:53:00.560200+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:12:28.921798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793129+0000","last_change":"2024-01-16T13:52:59.568183+0000","last_active":"2024-01-16T13:53:00.793129+0000","last_peered":"2024-01-16T13:53:00.793129+0000","last_clean":"2024-01-16T13:53:00.793129+0000","last_became_active":"2024-01-16T13:52:59.567833+0000","last_became_peered":"2024-01-16T13:52:59.567833+0000","last_unstale":"2024-01-16T13:53:00.793129+0000","last_undegraded":"2024-01-16T13:53:00.793129+0000","last_fullsized":"2024-01-16T13:53:00.793129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:08.471154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563286+0000","last_change":"2024-01-16T13:52:59.570264+0000","last_active":"2024-01-16T13:53:00.563286+0000","last_peered":"2024-01-16T13:53:00.563286+0000","last_clean":"2024-01-16T13:53:00.563286+0000","last_became_active":"2024-01-16T13:52:59.569849+0000","last_became_peered":"2024-01-16T13:52:59.569849+0000","last_unstale":"2024-01-16T13:53:00.563286+0000","last_undegraded":"2024-01-16T13:53:00.563286+0000","last_fullsized":"2024-01-16T13:53:00.563286+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:25.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791268+0000","last_change":"2024-01-16T13:52:59.567270+0000","last_active":"2024-01-16T13:53:00.791268+0000","last_peered":"2024-01-16T13:53:00.791268+0000","last_clean":"2024-01-16T13:53:00.791268+0000","last_became_active":"2024-01-16T13:52:59.567003+0000","last_became_peered":"2024-01-16T13:52:59.567003+0000","last_unstale":"2024-01-16T13:53:00.791268+0000","last_undegraded":"2024-01-16T13:53:00.791268+0000","last_fullsized":"2024-01-16T13:53:00.791268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:15.142935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609550+0000","last_change":"2024-01-16T13:52:59.568677+0000","last_active":"2024-01-16T13:53:00.609550+0000","last_peered":"2024-01-16T13:53:00.609550+0000","last_clean":"2024-01-16T13:53:00.609550+0000","last_became_active":"2024-01-16T13:52:59.568410+0000","last_became_peered":"2024-01-16T13:52:59.568410+0000","last_unstale":"2024-01-16T13:53:00.609550+0000","last_undegraded":"2024-01-16T13:53:00.609550+0000","last_fullsized":"2024-01-16T13:53:00.609550+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:21.972684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563074+0000","last_change":"2024-01-16T13:52:59.563121+0000","last_active":"2024-01-16T13:53:00.563074+0000","last_peered":"2024-01-16T13:53:00.563074+0000","last_clean":"2024-01-16T13:53:00.563074+0000","last_became_active":"2024-01-16T13:52:59.562812+0000","last_became_peered":"2024-01-16T13:52:59.562812+0000","last_unstale":"2024-01-16T13:53:00.563074+0000","last_undegraded":"2024-01-16T13:53:00.563074+0000","last_fullsized":"2024-01-16T13:53:00.563074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:18:18.563379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793121+0000","last_change":"2024-01-16T13:52:59.565538+0000","last_active":"2024-01-16T13:53:00.793121+0000","last_peered":"2024-01-16T13:53:00.793121+0000","last_clean":"2024-01-16T13:53:00.793121+0000","last_became_active":"2024-01-16T13:52:59.563776+0000","last_became_peered":"2024-01-16T13:52:59.563776+0000","last_unstale":"2024-01-16T13:53:00.793121+0000","last_undegraded":"2024-01-16T13:53:00.793121+0000","last_fullsized":"2024-01-16T13:53:00.793121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:34.906681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792450+0000","last_change":"2024-01-16T13:52:59.564678+0000","last_active":"2024-01-16T13:53:00.792450+0000","last_peered":"2024-01-16T13:53:00.792450+0000","last_clean":"2024-01-16T13:53:00.792450+0000","last_became_active":"2024-01-16T13:52:59.564382+0000","last_became_peered":"2024-01-16T13:52:59.564382+0000","last_unstale":"2024-01-16T13:53:00.792450+0000","last_undegraded":"2024-01-16T13:53:00.792450+0000","last_fullsized":"2024-01-16T13:53:00.792450+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:18:18.563379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609973+0000","last_change":"2024-01-16T13:52:59.570828+0000","last_active":"2024-01-16T13:53:00.609973+0000","last_peered":"2024-01-16T13:53:00.609973+0000","last_clean":"2024-01-16T13:53:00.609973+0000","last_became_active":"2024-01-16T13:52:59.570282+0000","last_became_peered":"2024-01-16T13:52:59.570282+0000","last_unstale":"2024-01-16T13:53:00.609973+0000","last_undegraded":"2024-01-16T13:53:00.609973+0000","last_fullsized":"2024-01-16T13:53:00.609973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:32.369974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609806+0000","last_change":"2024-01-16T13:52:59.570661+0000","last_active":"2024-01-16T13:53:00.609806+0000","last_peered":"2024-01-16T13:53:00.609806+0000","last_clean":"2024-01-16T13:53:00.609806+0000","last_became_active":"2024-01-16T13:52:59.570305+0000","last_became_peered":"2024-01-16T13:52:59.570305+0000","last_unstale":"2024-01-16T13:53:00.609806+0000","last_undegraded":"2024-01-16T13:53:00.609806+0000","last_fullsized":"2024-01-16T13:53:00.609806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:08.471154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791192+0000","last_change":"2024-01-16T13:52:59.567138+0000","last_active":"2024-01-16T13:53:00.791192+0000","last_peered":"2024-01-16T13:53:00.791192+0000","last_clean":"2024-01-16T13:53:00.791192+0000","last_became_active":"2024-01-16T13:52:59.566826+0000","last_became_peered":"2024-01-16T13:52:59.566826+0000","last_unstale":"2024-01-16T13:53:00.791192+0000","last_undegraded":"2024-01-16T13:53:00.791192+0000","last_fullsized":"2024-01-16T13:53:00.791192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:10:07.463675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792293+0000","last_change":"2024-01-16T13:52:59.569169+0000","last_active":"2024-01-16T13:53:00.792293+0000","last_peered":"2024-01-16T13:53:00.792293+0000","last_clean":"2024-01-16T13:53:00.792293+0000","last_became_active":"2024-01-16T13:52:59.568908+0000","last_became_peered":"2024-01-16T13:52:59.568908+0000","last_unstale":"2024-01-16T13:53:00.792293+0000","last_undegraded":"2024-01-16T13:53:00.792293+0000","last_fullsized":"2024-01-16T13:53:00.792293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:14.688323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793023+0000","last_change":"2024-01-16T13:52:59.554262+0000","last_active":"2024-01-16T13:53:00.793023+0000","last_peered":"2024-01-16T13:53:00.793023+0000","last_clean":"2024-01-16T13:53:00.793023+0000","last_became_active":"2024-01-16T13:52:59.553892+0000","last_became_peered":"2024-01-16T13:52:59.553892+0000","last_unstale":"2024-01-16T13:53:00.793023+0000","last_undegraded":"2024-01-16T13:53:00.793023+0000","last_fullsized":"2024-01-16T13:53:00.793023+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:00.590118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562661+0000","last_change":"2024-01-16T13:52:59.564937+0000","last_active":"2024-01-16T13:53:00.562661+0000","last_peered":"2024-01-16T13:53:00.562661+0000","last_clean":"2024-01-16T13:53:00.562661+0000","last_became_active":"2024-01-16T13:52:59.564618+0000","last_became_peered":"2024-01-16T13:52:59.564618+0000","last_unstale":"2024-01-16T13:53:00.562661+0000","last_undegraded":"2024-01-16T13:53:00.562661+0000","last_fullsized":"2024-01-16T13:53:00.562661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:50:00.627242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562884+0000","last_change":"2024-01-16T13:52:59.565960+0000","last_active":"2024-01-16T13:53:00.562884+0000","last_peered":"2024-01-16T13:53:00.562884+0000","last_clean":"2024-01-16T13:53:00.562884+0000","last_became_active":"2024-01-16T13:52:59.565667+0000","last_became_peered":"2024-01-16T13:52:59.565667+0000","last_unstale":"2024-01-16T13:53:00.562884+0000","last_undegraded":"2024-01-16T13:53:00.562884+0000","last_fullsized":"2024-01-16T13:53:00.562884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:32:24.125629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560360+0000","last_change":"2024-01-16T13:52:59.565505+0000","last_active":"2024-01-16T13:53:00.560360+0000","last_peered":"2024-01-16T13:53:00.560360+0000","last_clean":"2024-01-16T13:53:00.560360+0000","last_became_active":"2024-01-16T13:52:59.565212+0000","last_became_peered":"2024-01-16T13:52:59.565212+0000","last_unstale":"2024-01-16T13:53:00.560360+0000","last_undegraded":"2024-01-16T13:53:00.560360+0000","last_fullsized":"2024-01-16T13:53:00.560360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:34.336351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609949+0000","last_change":"2024-01-16T13:52:59.563999+0000","last_active":"2024-01-16T13:53:00.609949+0000","last_peered":"2024-01-16T13:53:00.609949+0000","last_clean":"2024-01-16T13:53:00.609949+0000","last_became_active":"2024-01-16T13:52:59.563738+0000","last_became_peered":"2024-01-16T13:52:59.563738+0000","last_unstale":"2024-01-16T13:53:00.609949+0000","last_undegraded":"2024-01-16T13:53:00.609949+0000","last_fullsized":"2024-01-16T13:53:00.609949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:34.382768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560375+0000","last_change":"2024-01-16T13:52:59.567123+0000","last_active":"2024-01-16T13:53:00.560375+0000","last_peered":"2024-01-16T13:53:00.560375+0000","last_clean":"2024-01-16T13:53:00.560375+0000","last_became_active":"2024-01-16T13:52:59.566557+0000","last_became_peered":"2024-01-16T13:52:59.566557+0000","last_unstale":"2024-01-16T13:53:00.560375+0000","last_undegraded":"2024-01-16T13:53:00.560375+0000","last_fullsized":"2024-01-16T13:53:00.560375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:53.788479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563411+0000","last_change":"2024-01-16T13:52:59.565257+0000","last_active":"2024-01-16T13:53:00.563411+0000","last_peered":"2024-01-16T13:53:00.563411+0000","last_clean":"2024-01-16T13:53:00.563411+0000","last_became_active":"2024-01-16T13:52:59.564963+0000","last_became_peered":"2024-01-16T13:52:59.564963+0000","last_unstale":"2024-01-16T13:53:00.563411+0000","last_undegraded":"2024-01-16T13:53:00.563411+0000","last_fullsized":"2024-01-16T13:53:00.563411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:34.906681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792449+0000","last_change":"2024-01-16T13:52:59.554147+0000","last_active":"2024-01-16T13:53:00.792449+0000","last_peered":"2024-01-16T13:53:00.792449+0000","last_clean":"2024-01-16T13:53:00.792449+0000","last_became_active":"2024-01-16T13:52:59.553868+0000","last_became_peered":"2024-01-16T13:52:59.553868+0000","last_unstale":"2024-01-16T13:53:00.792449+0000","last_undegraded":"2024-01-16T13:53:00.792449+0000","last_fullsized":"2024-01-16T13:53:00.792449+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:27.471640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792170+0000","last_change":"2024-01-16T13:52:59.567154+0000","last_active":"2024-01-16T13:53:00.792170+0000","last_peered":"2024-01-16T13:53:00.792170+0000","last_clean":"2024-01-16T13:53:00.792170+0000","last_became_active":"2024-01-16T13:52:59.566871+0000","last_became_peered":"2024-01-16T13:52:59.566871+0000","last_unstale":"2024-01-16T13:53:00.792170+0000","last_undegraded":"2024-01-16T13:53:00.792170+0000","last_fullsized":"2024-01-16T13:53:00.792170+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:02.720813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793277+0000","last_change":"2024-01-16T13:52:59.571801+0000","last_active":"2024-01-16T13:53:00.793277+0000","last_peered":"2024-01-16T13:53:00.793277+0000","last_clean":"2024-01-16T13:53:00.793277+0000","last_became_active":"2024-01-16T13:52:59.571468+0000","last_became_peered":"2024-01-16T13:52:59.571468+0000","last_unstale":"2024-01-16T13:53:00.793277+0000","last_undegraded":"2024-01-16T13:53:00.793277+0000","last_fullsized":"2024-01-16T13:53:00.793277+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:50:00.627242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792792+0000","last_change":"2024-01-16T13:52:59.565812+0000","last_active":"2024-01-16T13:53:00.792792+0000","last_peered":"2024-01-16T13:53:00.792792+0000","last_clean":"2024-01-16T13:53:00.792792+0000","last_became_active":"2024-01-16T13:52:59.565543+0000","last_became_peered":"2024-01-16T13:52:59.565543+0000","last_unstale":"2024-01-16T13:53:00.792792+0000","last_undegraded":"2024-01-16T13:53:00.792792+0000","last_fullsized":"2024-01-16T13:53:00.792792+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:25.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562806+0000","last_change":"2024-01-16T13:52:59.562924+0000","last_active":"2024-01-16T13:53:00.562806+0000","last_peered":"2024-01-16T13:53:00.562806+0000","last_clean":"2024-01-16T13:53:00.562806+0000","last_became_active":"2024-01-16T13:52:59.562629+0000","last_became_peered":"2024-01-16T13:52:59.562629+0000","last_unstale":"2024-01-16T13:53:00.562806+0000","last_undegraded":"2024-01-16T13:53:00.562806+0000","last_fullsized":"2024-01-16T13:53:00.562806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:05.512395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.572152+0000","last_change":"2024-01-16T13:52:58.559157+0000","last_active":"2024-01-16T13:52:59.572152+0000","last_peered":"2024-01-16T13:52:59.572152+0000","last_clean":"2024-01-16T13:52:59.572152+0000","last_became_active":"2024-01-16T13:52:58.558913+0000","last_became_peered":"2024-01-16T13:52:58.558913+0000","last_unstale":"2024-01-16T13:52:59.572152+0000","last_undegraded":"2024-01-16T13:52:59.572152+0000","last_fullsized":"2024-01-16T13:52:59.572152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:03.776676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792522+0000","last_change":"2024-01-16T13:52:59.563094+0000","last_active":"2024-01-16T13:53:00.792522+0000","last_peered":"2024-01-16T13:53:00.792522+0000","last_clean":"2024-01-16T13:53:00.792522+0000","last_became_active":"2024-01-16T13:52:59.562819+0000","last_became_peered":"2024-01-16T13:52:59.562819+0000","last_unstale":"2024-01-16T13:53:00.792522+0000","last_undegraded":"2024-01-16T13:53:00.792522+0000","last_fullsized":"2024-01-16T13:53:00.792522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:12:53.889907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562805+0000","last_change":"2024-01-16T13:52:58.557125+0000","last_active":"2024-01-16T13:53:00.562805+0000","last_peered":"2024-01-16T13:53:00.562805+0000","last_clean":"2024-01-16T13:53:00.562805+0000","last_became_active":"2024-01-16T13:52:58.556817+0000","last_became_peered":"2024-01-16T13:52:58.556817+0000","last_unstale":"2024-01-16T13:53:00.562805+0000","last_undegraded":"2024-01-16T13:53:00.562805+0000","last_fullsized":"2024-01-16T13:53:00.562805+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:54.995534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792954+0000","last_change":"2024-01-16T13:52:59.570308+0000","last_active":"2024-01-16T13:53:00.792954+0000","last_peered":"2024-01-16T13:53:00.792954+0000","last_clean":"2024-01-16T13:53:00.792954+0000","last_became_active":"2024-01-16T13:52:59.570029+0000","last_became_peered":"2024-01-16T13:52:59.570029+0000","last_unstale":"2024-01-16T13:53:00.792954+0000","last_undegraded":"2024-01-16T13:53:00.792954+0000","last_fullsized":"2024-01-16T13:53:00.792954+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:18:53.015251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.576132+0000","last_change":"2024-01-16T13:52:58.562496+0000","last_active":"2024-01-16T13:52:59.576132+0000","last_peered":"2024-01-16T13:52:59.576132+0000","last_clean":"2024-01-16T13:52:59.576132+0000","last_became_active":"2024-01-16T13:52:58.562231+0000","last_became_peered":"2024-01-16T13:52:58.562231+0000","last_unstale":"2024-01-16T13:52:59.576132+0000","last_undegraded":"2024-01-16T13:52:59.576132+0000","last_fullsized":"2024-01-16T13:52:59.576132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:16:59.587520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560532+0000","last_change":"2024-01-16T13:52:59.566968+0000","last_active":"2024-01-16T13:53:00.560532+0000","last_peered":"2024-01-16T13:53:00.560532+0000","last_clean":"2024-01-16T13:53:00.560532+0000","last_became_active":"2024-01-16T13:52:59.566268+0000","last_became_peered":"2024-01-16T13:52:59.566268+0000","last_unstale":"2024-01-16T13:53:00.560532+0000","last_undegraded":"2024-01-16T13:53:00.560532+0000","last_fullsized":"2024-01-16T13:53:00.560532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:02.720813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.572245+0000","last_change":"2024-01-16T13:52:58.562906+0000","last_active":"2024-01-16T13:52:59.572245+0000","last_peered":"2024-01-16T13:52:59.572245+0000","last_clean":"2024-01-16T13:52:59.572245+0000","last_became_active":"2024-01-16T13:52:58.562517+0000","last_became_peered":"2024-01-16T13:52:58.562517+0000","last_unstale":"2024-01-16T13:52:59.572245+0000","last_undegraded":"2024-01-16T13:52:59.572245+0000","last_fullsized":"2024-01-16T13:52:59.572245+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:37:27.869390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792327+0000","last_change":"2024-01-16T13:52:59.564090+0000","last_active":"2024-01-16T13:53:00.792327+0000","last_peered":"2024-01-16T13:53:00.792327+0000","last_clean":"2024-01-16T13:53:00.792327+0000","last_became_active":"2024-01-16T13:52:59.563810+0000","last_became_peered":"2024-01-16T13:52:59.563810+0000","last_unstale":"2024-01-16T13:53:00.792327+0000","last_undegraded":"2024-01-16T13:53:00.792327+0000","last_fullsized":"2024-01-16T13:53:00.792327+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:56.032353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.576352+0000","last_change":"2024-01-16T13:52:58.566758+0000","last_active":"2024-01-16T13:52:59.576352+0000","last_peered":"2024-01-16T13:52:59.576352+0000","last_clean":"2024-01-16T13:52:59.576352+0000","last_became_active":"2024-01-16T13:52:58.566541+0000","last_became_peered":"2024-01-16T13:52:58.566541+0000","last_unstale":"2024-01-16T13:52:59.576352+0000","last_undegraded":"2024-01-16T13:52:59.576352+0000","last_fullsized":"2024-01-16T13:52:59.576352+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:32.906444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563140+0000","last_change":"2024-01-16T13:52:59.566605+0000","last_active":"2024-01-16T13:53:00.563140+0000","last_peered":"2024-01-16T13:53:00.563140+0000","last_clean":"2024-01-16T13:53:00.563140+0000","last_became_active":"2024-01-16T13:52:59.566324+0000","last_became_peered":"2024-01-16T13:52:59.566324+0000","last_unstale":"2024-01-16T13:53:00.563140+0000","last_undegraded":"2024-01-16T13:53:00.563140+0000","last_fullsized":"2024-01-16T13:53:00.563140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:21.972684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.574564+0000","last_change":"2024-01-16T13:52:58.561808+0000","last_active":"2024-01-16T13:52:59.574564+0000","last_peered":"2024-01-16T13:52:59.574564+0000","last_clean":"2024-01-16T13:52:59.574564+0000","last_became_active":"2024-01-16T13:52:58.561490+0000","last_became_peered":"2024-01-16T13:52:58.561490+0000","last_unstale":"2024-01-16T13:52:59.574564+0000","last_undegraded":"2024-01-16T13:52:59.574564+0000","last_fullsized":"2024-01-16T13:52:59.574564+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:23.472571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792031+0000","last_change":"2024-01-16T13:52:59.559248+0000","last_active":"2024-01-16T13:53:00.792031+0000","last_peered":"2024-01-16T13:53:00.792031+0000","last_clean":"2024-01-16T13:53:00.792031+0000","last_became_active":"2024-01-16T13:52:59.558961+0000","last_became_peered":"2024-01-16T13:52:59.558961+0000","last_unstale":"2024-01-16T13:53:00.792031+0000","last_undegraded":"2024-01-16T13:53:00.792031+0000","last_fullsized":"2024-01-16T13:53:00.792031+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:50:00.627242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547589+0000","last_change":"2024-01-16T13:52:58.561453+0000","last_active":"2024-01-16T13:52:59.547589+0000","last_peered":"2024-01-16T13:52:59.547589+0000","last_clean":"2024-01-16T13:52:59.547589+0000","last_became_active":"2024-01-16T13:52:58.560956+0000","last_became_peered":"2024-01-16T13:52:58.560956+0000","last_unstale":"2024-01-16T13:52:59.547589+0000","last_undegraded":"2024-01-16T13:52:59.547589+0000","last_fullsized":"2024-01-16T13:52:59.547589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:03.019234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560198+0000","last_change":"2024-01-16T13:52:59.565380+0000","last_active":"2024-01-16T13:53:00.560198+0000","last_peered":"2024-01-16T13:53:00.560198+0000","last_clean":"2024-01-16T13:53:00.560198+0000","last_became_active":"2024-01-16T13:52:59.564986+0000","last_became_peered":"2024-01-16T13:52:59.564986+0000","last_unstale":"2024-01-16T13:53:00.560198+0000","last_undegraded":"2024-01-16T13:53:00.560198+0000","last_fullsized":"2024-01-16T13:53:00.560198+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:57.472694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.546031+0000","last_change":"2024-01-16T13:52:58.562274+0000","last_active":"2024-01-16T13:52:59.546031+0000","last_peered":"2024-01-16T13:52:59.546031+0000","last_clean":"2024-01-16T13:52:59.546031+0000","last_became_active":"2024-01-16T13:52:58.561978+0000","last_became_peered":"2024-01-16T13:52:58.561978+0000","last_unstale":"2024-01-16T13:52:59.546031+0000","last_undegraded":"2024-01-16T13:52:59.546031+0000","last_fullsized":"2024-01-16T13:52:59.546031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:36:13.374766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563291+0000","last_change":"2024-01-16T13:52:59.557554+0000","last_active":"2024-01-16T13:53:00.563291+0000","last_peered":"2024-01-16T13:53:00.563291+0000","last_clean":"2024-01-16T13:53:00.563291+0000","last_became_active":"2024-01-16T13:52:59.557271+0000","last_became_peered":"2024-01-16T13:52:59.557271+0000","last_unstale":"2024-01-16T13:53:00.563291+0000","last_undegraded":"2024-01-16T13:53:00.563291+0000","last_fullsized":"2024-01-16T13:53:00.563291+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:14.286572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.548340+0000","last_change":"2024-01-16T13:52:58.566147+0000","last_active":"2024-01-16T13:52:59.548340+0000","last_peered":"2024-01-16T13:52:59.548340+0000","last_clean":"2024-01-16T13:52:59.548340+0000","last_became_active":"2024-01-16T13:52:58.565861+0000","last_became_peered":"2024-01-16T13:52:58.565861+0000","last_unstale":"2024-01-16T13:52:59.548340+0000","last_undegraded":"2024-01-16T13:52:59.548340+0000","last_fullsized":"2024-01-16T13:52:59.548340+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:19.323195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609904+0000","last_change":"2024-01-16T13:52:59.569867+0000","last_active":"2024-01-16T13:53:00.609904+0000","last_peered":"2024-01-16T13:53:00.609904+0000","last_clean":"2024-01-16T13:53:00.609904+0000","last_became_active":"2024-01-16T13:52:59.569611+0000","last_became_peered":"2024-01-16T13:52:59.569611+0000","last_unstale":"2024-01-16T13:53:00.609904+0000","last_undegraded":"2024-01-16T13:53:00.609904+0000","last_fullsized":"2024-01-16T13:53:00.609904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:30.894670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562879+0000","last_change":"2024-01-16T13:52:58.562964+0000","last_active":"2024-01-16T13:53:00.562879+0000","last_peered":"2024-01-16T13:53:00.562879+0000","last_clean":"2024-01-16T13:53:00.562879+0000","last_became_active":"2024-01-16T13:52:58.562709+0000","last_became_peered":"2024-01-16T13:52:58.562709+0000","last_unstale":"2024-01-16T13:53:00.562879+0000","last_undegraded":"2024-01-16T13:53:00.562879+0000","last_fullsized":"2024-01-16T13:53:00.562879+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:38.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609659+0000","last_change":"2024-01-16T13:52:59.568844+0000","last_active":"2024-01-16T13:53:00.609659+0000","last_peered":"2024-01-16T13:53:00.609659+0000","last_clean":"2024-01-16T13:53:00.609659+0000","last_became_active":"2024-01-16T13:52:59.568587+0000","last_became_peered":"2024-01-16T13:52:59.568587+0000","last_unstale":"2024-01-16T13:53:00.609659+0000","last_undegraded":"2024-01-16T13:53:00.609659+0000","last_fullsized":"2024-01-16T13:53:00.609659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:51:25.977823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.563899+0000","last_change":"2024-01-16T13:52:58.556439+0000","last_active":"2024-01-16T13:52:59.563899+0000","last_peered":"2024-01-16T13:52:59.563899+0000","last_clean":"2024-01-16T13:52:59.563899+0000","last_became_active":"2024-01-16T13:52:58.556198+0000","last_became_peered":"2024-01-16T13:52:58.556198+0000","last_unstale":"2024-01-16T13:52:59.563899+0000","last_undegraded":"2024-01-16T13:52:59.563899+0000","last_fullsized":"2024-01-16T13:52:59.563899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:01.718215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563331+0000","last_change":"2024-01-16T13:52:59.570037+0000","last_active":"2024-01-16T13:53:00.563331+0000","last_peered":"2024-01-16T13:53:00.563331+0000","last_clean":"2024-01-16T13:53:00.563331+0000","last_became_active":"2024-01-16T13:52:59.569239+0000","last_became_peered":"2024-01-16T13:52:59.569239+0000","last_unstale":"2024-01-16T13:53:00.563331+0000","last_undegraded":"2024-01-16T13:53:00.563331+0000","last_fullsized":"2024-01-16T13:53:00.563331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:47:08.471154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.578050+0000","last_change":"2024-01-16T13:52:58.563418+0000","last_active":"2024-01-16T13:52:59.578050+0000","last_peered":"2024-01-16T13:52:59.578050+0000","last_clean":"2024-01-16T13:52:59.578050+0000","last_became_active":"2024-01-16T13:52:58.563149+0000","last_became_peered":"2024-01-16T13:52:58.563149+0000","last_unstale":"2024-01-16T13:52:59.578050+0000","last_undegraded":"2024-01-16T13:52:59.578050+0000","last_fullsized":"2024-01-16T13:52:59.578050+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:56.470097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.559947+0000","last_change":"2024-01-16T13:52:59.563254+0000","last_active":"2024-01-16T13:53:00.559947+0000","last_peered":"2024-01-16T13:53:00.559947+0000","last_clean":"2024-01-16T13:53:00.559947+0000","last_became_active":"2024-01-16T13:52:59.563001+0000","last_became_peered":"2024-01-16T13:52:59.563001+0000","last_unstale":"2024-01-16T13:53:00.559947+0000","last_undegraded":"2024-01-16T13:53:00.559947+0000","last_fullsized":"2024-01-16T13:53:00.559947+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:20.325793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.573503+0000","last_change":"2024-01-16T13:52:58.562574+0000","last_active":"2024-01-16T13:52:59.573503+0000","last_peered":"2024-01-16T13:52:59.573503+0000","last_clean":"2024-01-16T13:52:59.573503+0000","last_became_active":"2024-01-16T13:52:58.562303+0000","last_became_peered":"2024-01-16T13:52:58.562303+0000","last_unstale":"2024-01-16T13:52:59.573503+0000","last_undegraded":"2024-01-16T13:52:59.573503+0000","last_fullsized":"2024-01-16T13:52:59.573503+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:29.876112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609735+0000","last_change":"2024-01-16T13:52:59.566100+0000","last_active":"2024-01-16T13:53:00.609735+0000","last_peered":"2024-01-16T13:53:00.609735+0000","last_clean":"2024-01-16T13:53:00.609735+0000","last_became_active":"2024-01-16T13:52:59.565846+0000","last_became_peered":"2024-01-16T13:52:59.565846+0000","last_unstale":"2024-01-16T13:53:00.609735+0000","last_undegraded":"2024-01-16T13:53:00.609735+0000","last_fullsized":"2024-01-16T13:53:00.609735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:14.286572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.554048+0000","last_change":"2024-01-16T13:52:58.559322+0000","last_active":"2024-01-16T13:52:59.554048+0000","last_peered":"2024-01-16T13:52:59.554048+0000","last_clean":"2024-01-16T13:52:59.554048+0000","last_became_active":"2024-01-16T13:52:58.558802+0000","last_became_peered":"2024-01-16T13:52:58.558802+0000","last_unstale":"2024-01-16T13:52:59.554048+0000","last_undegraded":"2024-01-16T13:52:59.554048+0000","last_fullsized":"2024-01-16T13:52:59.554048+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:38.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792614+0000","last_change":"2024-01-16T13:52:59.564532+0000","last_active":"2024-01-16T13:53:00.792614+0000","last_peered":"2024-01-16T13:53:00.792614+0000","last_clean":"2024-01-16T13:53:00.792614+0000","last_became_active":"2024-01-16T13:52:59.564247+0000","last_became_peered":"2024-01-16T13:52:59.564247+0000","last_unstale":"2024-01-16T13:53:00.792614+0000","last_undegraded":"2024-01-16T13:53:00.792614+0000","last_fullsized":"2024-01-16T13:53:00.792614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:20:21.972684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547723+0000","last_change":"2024-01-16T13:52:58.561629+0000","last_active":"2024-01-16T13:52:59.547723+0000","last_peered":"2024-01-16T13:52:59.547723+0000","last_clean":"2024-01-16T13:52:59.547723+0000","last_became_active":"2024-01-16T13:52:58.561193+0000","last_became_peered":"2024-01-16T13:52:58.561193+0000","last_unstale":"2024-01-16T13:52:59.547723+0000","last_undegraded":"2024-01-16T13:52:59.547723+0000","last_fullsized":"2024-01-16T13:52:59.547723+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:49:59.624644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792417+0000","last_change":"2024-01-16T13:52:59.564246+0000","last_active":"2024-01-16T13:53:00.792417+0000","last_peered":"2024-01-16T13:53:00.792417+0000","last_clean":"2024-01-16T13:53:00.792417+0000","last_became_active":"2024-01-16T13:52:59.563968+0000","last_became_peered":"2024-01-16T13:52:59.563968+0000","last_unstale":"2024-01-16T13:53:00.792417+0000","last_undegraded":"2024-01-16T13:53:00.792417+0000","last_fullsized":"2024-01-16T13:53:00.792417+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:38.154103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.577048+0000","last_change":"2024-01-16T13:52:58.566564+0000","last_active":"2024-01-16T13:52:59.577048+0000","last_peered":"2024-01-16T13:52:59.577048+0000","last_clean":"2024-01-16T13:52:59.577048+0000","last_became_active":"2024-01-16T13:52:58.566241+0000","last_became_peered":"2024-01-16T13:52:58.566241+0000","last_unstale":"2024-01-16T13:52:59.577048+0000","last_undegraded":"2024-01-16T13:52:59.577048+0000","last_fullsized":"2024-01-16T13:52:59.577048+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:13.685725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.793086+0000","last_change":"2024-01-16T13:52:59.557019+0000","last_active":"2024-01-16T13:53:00.793086+0000","last_peered":"2024-01-16T13:53:00.793086+0000","last_clean":"2024-01-16T13:53:00.793086+0000","last_became_active":"2024-01-16T13:52:59.556754+0000","last_became_peered":"2024-01-16T13:52:59.556754+0000","last_unstale":"2024-01-16T13:53:00.793086+0000","last_undegraded":"2024-01-16T13:53:00.793086+0000","last_fullsized":"2024-01-16T13:53:00.793086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:57.472694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.576731+0000","last_change":"2024-01-16T13:52:58.561600+0000","last_active":"2024-01-16T13:52:59.576731+0000","last_peered":"2024-01-16T13:52:59.576731+0000","last_clean":"2024-01-16T13:52:59.576731+0000","last_became_active":"2024-01-16T13:52:58.561333+0000","last_became_peered":"2024-01-16T13:52:58.561333+0000","last_unstale":"2024-01-16T13:52:59.576731+0000","last_undegraded":"2024-01-16T13:52:59.576731+0000","last_fullsized":"2024-01-16T13:52:59.576731+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:04.509797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563723+0000","last_change":"2024-01-16T13:52:59.554017+0000","last_active":"2024-01-16T13:53:00.563723+0000","last_peered":"2024-01-16T13:53:00.563723+0000","last_clean":"2024-01-16T13:53:00.563723+0000","last_became_active":"2024-01-16T13:52:59.553676+0000","last_became_peered":"2024-01-16T13:52:59.553676+0000","last_unstale":"2024-01-16T13:53:00.563723+0000","last_undegraded":"2024-01-16T13:53:00.563723+0000","last_fullsized":"2024-01-16T13:53:00.563723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:34.382768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.568454+0000","last_change":"2024-01-16T13:52:58.562411+0000","last_active":"2024-01-16T13:52:59.568454+0000","last_peered":"2024-01-16T13:52:59.568454+0000","last_clean":"2024-01-16T13:52:59.568454+0000","last_became_active":"2024-01-16T13:52:58.562031+0000","last_became_peered":"2024-01-16T13:52:58.562031+0000","last_unstale":"2024-01-16T13:52:59.568454+0000","last_undegraded":"2024-01-16T13:52:59.568454+0000","last_fullsized":"2024-01-16T13:52:59.568454+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:14.140338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.560237+0000","last_change":"2024-01-16T13:52:59.564287+0000","last_active":"2024-01-16T13:53:00.560237+0000","last_peered":"2024-01-16T13:53:00.560237+0000","last_clean":"2024-01-16T13:53:00.560237+0000","last_became_active":"2024-01-16T13:52:59.564029+0000","last_became_peered":"2024-01-16T13:52:59.564029+0000","last_unstale":"2024-01-16T13:53:00.560237+0000","last_undegraded":"2024-01-16T13:53:00.560237+0000","last_fullsized":"2024-01-16T13:53:00.560237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:17:00.590118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.570508+0000","last_change":"2024-01-16T13:52:58.562668+0000","last_active":"2024-01-16T13:52:59.570508+0000","last_peered":"2024-01-16T13:52:59.570508+0000","last_clean":"2024-01-16T13:52:59.570508+0000","last_became_active":"2024-01-16T13:52:58.562432+0000","last_became_peered":"2024-01-16T13:52:58.562432+0000","last_unstale":"2024-01-16T13:52:59.570508+0000","last_undegraded":"2024-01-16T13:52:59.570508+0000","last_fullsized":"2024-01-16T13:52:59.570508+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:04.946684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791140+0000","last_change":"2024-01-16T13:52:59.566650+0000","last_active":"2024-01-16T13:53:00.791140+0000","last_peered":"2024-01-16T13:53:00.791140+0000","last_clean":"2024-01-16T13:53:00.791140+0000","last_became_active":"2024-01-16T13:52:59.566395+0000","last_became_peered":"2024-01-16T13:52:59.566395+0000","last_unstale":"2024-01-16T13:53:00.791140+0000","last_undegraded":"2024-01-16T13:53:00.791140+0000","last_fullsized":"2024-01-16T13:53:00.791140+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:25.689016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.546600+0000","last_change":"2024-01-16T13:52:58.566549+0000","last_active":"2024-01-16T13:52:59.546600+0000","last_peered":"2024-01-16T13:52:59.546600+0000","last_clean":"2024-01-16T13:52:59.546600+0000","last_became_active":"2024-01-16T13:52:58.566269+0000","last_became_peered":"2024-01-16T13:52:58.566269+0000","last_unstale":"2024-01-16T13:52:59.546600+0000","last_undegraded":"2024-01-16T13:52:59.546600+0000","last_fullsized":"2024-01-16T13:52:59.546600+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:49.846088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791078+0000","last_change":"2024-01-16T13:52:59.566738+0000","last_active":"2024-01-16T13:53:00.791078+0000","last_peered":"2024-01-16T13:53:00.791078+0000","last_clean":"2024-01-16T13:53:00.791078+0000","last_became_active":"2024-01-16T13:52:59.566462+0000","last_became_peered":"2024-01-16T13:52:59.566462+0000","last_unstale":"2024-01-16T13:53:00.791078+0000","last_undegraded":"2024-01-16T13:53:00.791078+0000","last_fullsized":"2024-01-16T13:53:00.791078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:36:14.377364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562804+0000","last_change":"2024-01-16T13:52:58.559782+0000","last_active":"2024-01-16T13:53:00.562804+0000","last_peered":"2024-01-16T13:53:00.562804+0000","last_clean":"2024-01-16T13:53:00.562804+0000","last_became_active":"2024-01-16T13:52:58.559472+0000","last_became_peered":"2024-01-16T13:52:58.559472+0000","last_unstale":"2024-01-16T13:53:00.562804+0000","last_undegraded":"2024-01-16T13:53:00.562804+0000","last_fullsized":"2024-01-16T13:53:00.562804+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:16:37.151505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791132+0000","last_change":"2024-01-16T13:52:59.569265+0000","last_active":"2024-01-16T13:53:00.791132+0000","last_peered":"2024-01-16T13:53:00.791132+0000","last_clean":"2024-01-16T13:53:00.791132+0000","last_became_active":"2024-01-16T13:52:59.569074+0000","last_became_peered":"2024-01-16T13:52:59.569074+0000","last_unstale":"2024-01-16T13:53:00.791132+0000","last_undegraded":"2024-01-16T13:53:00.791132+0000","last_fullsized":"2024-01-16T13:53:00.791132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:05.949282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.553273+0000","last_change":"2024-01-16T13:52:58.563508+0000","last_active":"2024-01-16T13:52:59.553273+0000","last_peered":"2024-01-16T13:52:59.553273+0000","last_clean":"2024-01-16T13:52:59.553273+0000","last_became_active":"2024-01-16T13:52:58.562973+0000","last_became_peered":"2024-01-16T13:52:58.562973+0000","last_unstale":"2024-01-16T13:52:59.553273+0000","last_undegraded":"2024-01-16T13:52:59.553273+0000","last_fullsized":"2024-01-16T13:52:59.553273+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:53:38.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791082+0000","last_change":"2024-01-16T13:52:59.569332+0000","last_active":"2024-01-16T13:53:00.791082+0000","last_peered":"2024-01-16T13:53:00.791082+0000","last_clean":"2024-01-16T13:53:00.791082+0000","last_became_active":"2024-01-16T13:52:59.569036+0000","last_became_peered":"2024-01-16T13:52:59.569036+0000","last_unstale":"2024-01-16T13:53:00.791082+0000","last_undegraded":"2024-01-16T13:53:00.791082+0000","last_fullsized":"2024-01-16T13:53:00.791082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:24.475169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.546853+0000","last_change":"2024-01-16T13:52:58.563616+0000","last_active":"2024-01-16T13:52:59.546853+0000","last_peered":"2024-01-16T13:52:59.546853+0000","last_clean":"2024-01-16T13:52:59.546853+0000","last_became_active":"2024-01-16T13:52:58.563302+0000","last_became_peered":"2024-01-16T13:52:58.563302+0000","last_unstale":"2024-01-16T13:52:59.546853+0000","last_undegraded":"2024-01-16T13:52:59.546853+0000","last_fullsized":"2024-01-16T13:52:59.546853+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:24.686418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792793+0000","last_change":"2024-01-16T13:52:59.567226+0000","last_active":"2024-01-16T13:53:00.792793+0000","last_peered":"2024-01-16T13:53:00.792793+0000","last_clean":"2024-01-16T13:53:00.792793+0000","last_became_active":"2024-01-16T13:52:59.566659+0000","last_became_peered":"2024-01-16T13:52:59.566659+0000","last_unstale":"2024-01-16T13:53:00.792793+0000","last_undegraded":"2024-01-16T13:53:00.792793+0000","last_fullsized":"2024-01-16T13:53:00.792793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:14.286572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.548876+0000","last_change":"2024-01-16T13:52:58.559155+0000","last_active":"2024-01-16T13:52:59.548876+0000","last_peered":"2024-01-16T13:52:59.548876+0000","last_clean":"2024-01-16T13:52:59.548876+0000","last_became_active":"2024-01-16T13:52:58.558863+0000","last_became_peered":"2024-01-16T13:52:58.558863+0000","last_unstale":"2024-01-16T13:52:59.548876+0000","last_undegraded":"2024-01-16T13:52:59.548876+0000","last_fullsized":"2024-01-16T13:52:59.548876+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:56.470097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.791138+0000","last_change":"2024-01-16T13:52:59.567708+0000","last_active":"2024-01-16T13:53:00.791138+0000","last_peered":"2024-01-16T13:53:00.791138+0000","last_clean":"2024-01-16T13:53:00.791138+0000","last_became_active":"2024-01-16T13:52:59.567461+0000","last_became_peered":"2024-01-16T13:52:59.567461+0000","last_unstale":"2024-01-16T13:53:00.791138+0000","last_undegraded":"2024-01-16T13:53:00.791138+0000","last_fullsized":"2024-01-16T13:53:00.791138+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:50.848686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.572479+0000","last_change":"2024-01-16T13:52:58.559075+0000","last_active":"2024-01-16T13:52:59.572479+0000","last_peered":"2024-01-16T13:52:59.572479+0000","last_clean":"2024-01-16T13:52:59.572479+0000","last_became_active":"2024-01-16T13:52:58.558722+0000","last_became_peered":"2024-01-16T13:52:58.558722+0000","last_unstale":"2024-01-16T13:52:59.572479+0000","last_undegraded":"2024-01-16T13:52:59.572479+0000","last_fullsized":"2024-01-16T13:52:59.572479+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:13.685725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.563669+0000","last_change":"2024-01-16T13:52:59.569889+0000","last_active":"2024-01-16T13:53:00.563669+0000","last_peered":"2024-01-16T13:53:00.563669+0000","last_clean":"2024-01-16T13:53:00.563669+0000","last_became_active":"2024-01-16T13:52:59.569282+0000","last_became_peered":"2024-01-16T13:52:59.569282+0000","last_unstale":"2024-01-16T13:53:00.563669+0000","last_undegraded":"2024-01-16T13:53:00.563669+0000","last_fullsized":"2024-01-16T13:53:00.563669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:30.894670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:01.101751+0000","last_change":"2024-01-16T13:52:58.559563+0000","last_active":"2024-01-16T13:53:01.101751+0000","last_peered":"2024-01-16T13:53:01.101751+0000","last_clean":"2024-01-16T13:53:01.101751+0000","last_became_active":"2024-01-16T13:52:58.559233+0000","last_became_peered":"2024-01-16T13:52:58.559233+0000","last_unstale":"2024-01-16T13:53:01.101751+0000","last_undegraded":"2024-01-16T13:53:01.101751+0000","last_fullsized":"2024-01-16T13:53:01.101751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:03.776676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562783+0000","last_change":"2024-01-16T13:52:59.572336+0000","last_active":"2024-01-16T13:53:00.562783+0000","last_peered":"2024-01-16T13:53:00.562783+0000","last_clean":"2024-01-16T13:53:00.562783+0000","last_became_active":"2024-01-16T13:52:59.572074+0000","last_became_peered":"2024-01-16T13:52:59.572074+0000","last_unstale":"2024-01-16T13:53:00.562783+0000","last_undegraded":"2024-01-16T13:53:00.562783+0000","last_fullsized":"2024-01-16T13:53:00.562783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:18:14.688323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.554523+0000","last_change":"2024-01-16T13:52:58.558949+0000","last_active":"2024-01-16T13:52:59.554523+0000","last_peered":"2024-01-16T13:52:59.554523+0000","last_clean":"2024-01-16T13:52:59.554523+0000","last_became_active":"2024-01-16T13:52:58.558648+0000","last_became_peered":"2024-01-16T13:52:58.558648+0000","last_unstale":"2024-01-16T13:52:59.554523+0000","last_undegraded":"2024-01-16T13:52:59.554523+0000","last_fullsized":"2024-01-16T13:52:59.554523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:21:32.906444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792034+0000","last_change":"2024-01-16T13:52:59.566492+0000","last_active":"2024-01-16T13:53:00.792034+0000","last_peered":"2024-01-16T13:53:00.792034+0000","last_clean":"2024-01-16T13:53:00.792034+0000","last_became_active":"2024-01-16T13:52:59.566221+0000","last_became_peered":"2024-01-16T13:52:59.566221+0000","last_unstale":"2024-01-16T13:53:00.792034+0000","last_undegraded":"2024-01-16T13:53:00.792034+0000","last_fullsized":"2024-01-16T13:53:00.792034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:04.021832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562737+0000","last_change":"2024-01-16T13:52:58.553716+0000","last_active":"2024-01-16T13:53:00.562737+0000","last_peered":"2024-01-16T13:53:00.562737+0000","last_clean":"2024-01-16T13:53:00.562737+0000","last_became_active":"2024-01-16T13:52:58.553078+0000","last_became_peered":"2024-01-16T13:52:58.553078+0000","last_unstale":"2024-01-16T13:53:00.562737+0000","last_undegraded":"2024-01-16T13:53:00.562737+0000","last_fullsized":"2024-01-16T13:53:00.562737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:29.876112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.562911+0000","last_change":"2024-01-16T13:52:59.566180+0000","last_active":"2024-01-16T13:53:00.562911+0000","last_peered":"2024-01-16T13:53:00.562911+0000","last_clean":"2024-01-16T13:53:00.562911+0000","last_became_active":"2024-01-16T13:52:59.565847+0000","last_became_peered":"2024-01-16T13:52:59.565847+0000","last_unstale":"2024-01-16T13:53:00.562911+0000","last_undegraded":"2024-01-16T13:53:00.562911+0000","last_fullsized":"2024-01-16T13:53:00.562911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:27.471640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.555191+0000","last_change":"2024-01-16T13:52:58.559484+0000","last_active":"2024-01-16T13:52:59.555191+0000","last_peered":"2024-01-16T13:52:59.555191+0000","last_clean":"2024-01-16T13:52:59.555191+0000","last_became_active":"2024-01-16T13:52:58.559084+0000","last_became_peered":"2024-01-16T13:52:58.559084+0000","last_unstale":"2024-01-16T13:52:59.555191+0000","last_undegraded":"2024-01-16T13:52:59.555191+0000","last_fullsized":"2024-01-16T13:52:59.555191+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:12:52.887309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.609403+0000","last_change":"2024-01-16T13:52:59.564927+0000","last_active":"2024-01-16T13:53:00.609403+0000","last_peered":"2024-01-16T13:53:00.609403+0000","last_clean":"2024-01-16T13:53:00.609403+0000","last_became_active":"2024-01-16T13:52:59.564679+0000","last_became_peered":"2024-01-16T13:52:59.564679+0000","last_unstale":"2024-01-16T13:53:00.609403+0000","last_undegraded":"2024-01-16T13:53:00.609403+0000","last_fullsized":"2024-01-16T13:53:00.609403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:18:18.563379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547015+0000","last_change":"2024-01-16T13:52:58.559066+0000","last_active":"2024-01-16T13:52:59.547015+0000","last_peered":"2024-01-16T13:52:59.547015+0000","last_clean":"2024-01-16T13:52:59.547015+0000","last_became_active":"2024-01-16T13:52:58.558776+0000","last_became_peered":"2024-01-16T13:52:58.558776+0000","last_unstale":"2024-01-16T13:52:59.547015+0000","last_undegraded":"2024-01-16T13:52:59.547015+0000","last_fullsized":"2024-01-16T13:52:59.547015+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:54.995534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792349+0000","last_change":"2024-01-16T13:52:59.553174+0000","last_active":"2024-01-16T13:53:00.792349+0000","last_peered":"2024-01-16T13:53:00.792349+0000","last_clean":"2024-01-16T13:53:00.792349+0000","last_became_active":"2024-01-16T13:52:59.552849+0000","last_became_peered":"2024-01-16T13:52:59.552849+0000","last_unstale":"2024-01-16T13:53:00.792349+0000","last_undegraded":"2024-01-16T13:53:00.792349+0000","last_fullsized":"2024-01-16T13:53:00.792349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:05.512395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.547961+0000","last_change":"2024-01-16T13:52:58.561252+0000","last_active":"2024-01-16T13:52:59.547961+0000","last_peered":"2024-01-16T13:52:59.547961+0000","last_clean":"2024-01-16T13:52:59.547961+0000","last_became_active":"2024-01-16T13:52:58.560969+0000","last_became_peered":"2024-01-16T13:52:58.560969+0000","last_unstale":"2024-01-16T13:52:59.547961+0000","last_undegraded":"2024-01-16T13:52:59.547961+0000","last_fullsized":"2024-01-16T13:52:59.547961+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T14:37:04.509797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-16T13:53:00.792085+0000","last_change":"2024-01-16T13:52:59.569488+0000","last_active":"2024-01-16T13:53:00.792085+0000","last_peered":"2024-01-16T13:53:00.792085+0000","last_clean":"2024-01-16T13:53:00.792085+0000","last_became_active":"2024-01-16T13:52:59.569033+0000","last_became_peered":"2024-01-16T13:52:59.569033+0000","last_unstale":"2024-01-16T13:53:00.792085+0000","last_undegraded":"2024-01-16T13:53:00.792085+0000","last_fullsized":"2024-01-16T13:53:00.792085+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:58.534818+0000","last_clean_scrub_stamp":"2024-01-16T13:52:58.534818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:53.788479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.552892+0000","last_change":"2024-01-16T13:52:58.562363+0000","last_active":"2024-01-16T13:52:59.552892+0000","last_peered":"2024-01-16T13:52:59.552892+0000","last_clean":"2024-01-16T13:52:59.552892+0000","last_became_active":"2024-01-16T13:52:58.562073+0000","last_became_peered":"2024-01-16T13:52:58.562073+0000","last_unstale":"2024-01-16T13:52:59.552892+0000","last_undegraded":"2024-01-16T13:52:59.552892+0000","last_fullsized":"2024-01-16T13:52:59.552892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:52:57.532220+0000","last_clean_scrub_stamp":"2024-01-16T13:52:57.532220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:38:54.995534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":173,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-16T13:52:59.552274+0000","last_change":"2024-01-16T13:52:00.480913+0000","last_active":"2024-01-16T13:52:59.552274+0000","last_peered":"2024-01-16T13:52:59.552274+0000","last_clean":"2024-01-16T13:52:59.552274+0000","last_became_active":"2024-01-16T13:52:00.479945+0000","last_became_peered":"2024-01-16T13:52:00.479945+0000","last_unstale":"2024-01-16T13:52:59.552274+0000","last_undegraded":"2024-01-16T13:52:59.552274+0000","last_fullsized":"2024-01-16T13:52:59.552274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T13:51:58.854768+0000","last_clean_scrub_stamp":"2024-01-16T13:51:58.854768+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":734546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.497}]},{"osd":3,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.73199999999999998}]},{"osd":5,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.77600000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.83099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55100000000000005}]},{"osd":1,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.49199999999999999}]},{"osd":3,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40600000000000003}]},{"osd":4,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.42299999999999999}]},{"osd":5,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Jan 16 13:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.53700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":145722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.374,"5min":0.374,"15min":0.374},"last":0.318}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.58399999999999996}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.55500000000000005}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.746},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56399999999999995}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.79200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":734456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.45200000000000001}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.39900000000000002}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.49399999999999999}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63900000000000001}]},{"osd":5,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.65200000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.621}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.72199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":144158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.34899999999999998}]},{"osd":1,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.48199999999999998}]},{"osd":3,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.373}]},{"osd":4,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.52200000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.505},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64400000000000002}]},{"osd":7,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.57099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":145722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.377}]},{"osd":1,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.34799999999999998}]},{"osd":4,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.495}]},{"osd":5,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58099999999999996}]},{"osd":6,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.505}]},{"osd":7,"last update":"Tue Jan 16 13:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.59899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.72399999999999998}]},{"osd":1,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":6.8289999999999997,"5min":6.8289999999999997,"15min":6.8289999999999997},"last":0.70699999999999996}]},{"osd":2,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.843,"5min":5.843,"15min":5.843},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":6.3079999999999998,"5min":6.3079999999999998,"15min":6.3079999999999998},"last":0.61399999999999999}]},{"osd":3,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":6.7690000000000001,"5min":6.7690000000000001,"15min":6.7690000000000001},"last":0.625},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.52200000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.433}]},{"osd":7,"last update":"Tue Jan 16 13:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.55700000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":144760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.63800000000000001}]},{"osd":1,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.624}]},{"osd":3,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.66600000000000004}]},{"osd":4,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.40300000000000002}]},{"osd":6,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Tue Jan 16 13:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T13:53:08.552 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T13:53:08.552 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T13:53:08.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.734+0000 7fd8ebe9c700 1 Processor -- start 2024-01-16T13:53:08.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.734+0000 7fd8ebe9c700 1 -- start start 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ebe9c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406f680 0x7fd8e40afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ebe9c700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8e406e420 0x7fd8e406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ebe9c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e406fb00 con 0x7fd8e415e110 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ebe9c700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e40b0070 con 0x7fd8e406e420 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ebe9c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e406d980 con 0x7fd8e406f680 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ea439700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8e406e420 0x7fd8e406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ea439700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd8e406e420 0x7fd8e406e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50042/0 (socket says 172.21.15.113:50042) 2024-01-16T13:53:08.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.735+0000 7fd8ea439700 1 -- 172.21.15.113:0/2848774557 learned_addr learned my addr 172.21.15.113:0/2848774557 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8e9437700 1 --2- 172.21.15.113:0/2848774557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406f680 0x7fd8e40afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8ea439700 1 -- 172.21.15.113:0/2848774557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406f680 msgr2=0x7fd8e40afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8ea439700 1 --2- 172.21.15.113:0/2848774557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406f680 0x7fd8e40afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8ea439700 1 -- 172.21.15.113:0/2848774557 >> v1:172.21.15.124:6789/0 conn(0x7fd8e415e110 legacy=0x7fd8e406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8ea439700 1 -- 172.21.15.113:0/2848774557 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e40b0260 con 0x7fd8e406e420 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8ea439700 1 --2- 172.21.15.113:0/2848774557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406e420 0x7fd8e406e8b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009e50 tx=0x7fd8dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2309dc922cb91f71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8e8c36700 1 -- 172.21.15.113:0/2848774557 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc00e070 con 0x7fd8e406e420 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8e8c36700 1 -- 172.21.15.113:0/2848774557 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc00b9c0 con 0x7fd8e406e420 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.736+0000 7fd8e8c36700 1 -- 172.21.15.113:0/2848774557 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc00bd70 con 0x7fd8e406e420 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/2848774557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406e420 msgr2=0x7fd8e406e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/2848774557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406e420 0x7fd8e406e8b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009e50 tx=0x7fd8dc00b330 comp rx=0 tx=0).stop 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/2848774557 shutdown_connections 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/2848774557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406e420 0x7fd8e406e8b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/2848774557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406f680 0x7fd8e40afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/2848774557 >> 172.21.15.113:0/2848774557 conn(0x7fd8e40b3250 msgr2=0x7fd8e40d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:08.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/2848774557 shutdown_connections 2024-01-16T13:53:08.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.737+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/2848774557 wait complete. 2024-01-16T13:53:08.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.738+0000 7fd8ebe9c700 1 Processor -- start 2024-01-16T13:53:08.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.738+0000 7fd8ebe9c700 1 -- start start 2024-01-16T13:53:08.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406e420 0x7fd8e40c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 0x7fd8e414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8e415e110 0x7fd8e40c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40c6a50 con 0x7fd8e406f680 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40c5b60 con 0x7fd8e415e110 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40c5e40 con 0x7fd8e406e420 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 0x7fd8e414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 0x7fd8e414b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50054/0 (socket says 172.21.15.113:50054) 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 -- 172.21.15.113:0/1580938990 learned_addr learned my addr 172.21.15.113:0/1580938990 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ea439700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8e415e110 0x7fd8e40c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9c38700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406e420 0x7fd8e40c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 -- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406e420 msgr2=0x7fd8e40c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406e420 0x7fd8e40c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 -- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8e415e110 msgr2=0x7fd8e40c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8e415e110 0x7fd8e40c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 -- 172.21.15.113:0/1580938990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc009b30 con 0x7fd8e406f680 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8e9437700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 0x7fd8e414b5f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e810 tx=0x7fd8e000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:08.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000c070 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0013610 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e40c6120 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00139c0 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.739+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e40c66e0 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.740+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e4069020 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.741+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd8e0013b20 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.741+0000 7fd8da7fc700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8c806b350 0x7fd8c806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.741+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8e00898a0 con 0x7fd8e406f680 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.741+0000 7fd8e9c38700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8c806b350 0x7fd8c806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:08.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.742+0000 7fd8e9c38700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8c806b350 0x7fd8c806d810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40d0420 tx=0x7fd8d4009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:08.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:08.746+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8e0054720 con 0x7fd8e406f680 2024-01-16T13:53:08.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.014+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd8e40c6b90 con 0x7fd8e406f680 2024-01-16T13:53:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.018+0000 7fd8da7fc700 1 -- 172.21.15.113:0/1580938990 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd8e0057d40 con 0x7fd8e406f680 2024-01-16T13:53:09.000 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2024-01-16T13:53:09.000 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQDRiaZlwFPiGRAAc/ztvEiVlF5DA4U3NZB3Og== 2024-01-16T13:53:09.000 INFO:teuthology.orchestra.run.smithi113.stdout: caps mds = "allow" 2024-01-16T13:53:09.000 INFO:teuthology.orchestra.run.smithi113.stdout: caps mon = "allow r" 2024-01-16T13:53:09.000 INFO:teuthology.orchestra.run.smithi113.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T13:53:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8c806b350 msgr2=0x7fd8c806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8c806b350 0x7fd8c806d810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40d0420 tx=0x7fd8d4009040 comp rx=0 tx=0).stop 2024-01-16T13:53:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 msgr2=0x7fd8e414b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 0x7fd8e414b5f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e810 tx=0x7fd8e000eb20 comp rx=0 tx=0).stop 2024-01-16T13:53:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 shutdown_connections 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fd8c806b350 0x7fd8c806d810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8e415e110 0x7fd8e40c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd8e406e420 0x7fd8e40c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 --2- 172.21.15.113:0/1580938990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8e406f680 0x7fd8e414b5f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 >> 172.21.15.113:0/1580938990 conn(0x7fd8e40b3250 msgr2=0x7fd8e40b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 shutdown_connections 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.020+0000 7fd8ebe9c700 1 -- 172.21.15.113:0/1580938990 wait complete. 2024-01-16T13:53:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:updated caps for client.0 2024-01-16T13:53:09.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T13:53:09.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.153+0000 7f6473931700 1 Processor -- start 2024-01-16T13:53:09.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.154+0000 7f6473931700 1 -- start start 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.154+0000 7f6473931700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f646c06c470 0x7f646c0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.154+0000 7f6473931700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c0b42d0 0x7f646c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.154+0000 7f6473931700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c06b3d0 con 0x7f646c073f90 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.154+0000 7f6473931700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c06b6b0 con 0x7f646c06c470 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6473931700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c06b990 con 0x7f646c0b42d0 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f646c06c470 0x7f646c0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f646c06c470 0x7f646c0b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50068/0 (socket says 172.21.15.113:50068) 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 -- 172.21.15.113:0/3020710518 learned_addr learned my addr 172.21.15.113:0/3020710518 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f64716cd700 1 --1- 172.21.15.113:0/3020710518 >> v1:172.21.15.113:6789/0 conn(0x7f646c073f90 0x7f646c074370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:45256/0 (socket says 172.21.15.113:45256) 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 -- 172.21.15.113:0/3020710518 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c0b42d0 msgr2=0x7f646c06dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 --2- 172.21.15.113:0/3020710518 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c0b42d0 0x7f646c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 -- 172.21.15.113:0/3020710518 >> v1:172.21.15.113:6789/0 conn(0x7f646c073f90 legacy=0x7f646c074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 -- 172.21.15.113:0/3020710518 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c06bc70 con 0x7f646c06c470 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6463fff700 1 -- 172.21.15.113:0/3020710518 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186720392 0 0) 0x7f646c06b3d0 con 0x7f646c073f90 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6470ecc700 1 --2- 172.21.15.113:0/3020710518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c06c470 0x7f646c0b3d90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f64680098f0 tx=0x7f6468009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edb464160579dcd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6463fff700 1 -- 172.21.15.113:0/3020710518 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800b8a0 con 0x7f646c06c470 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6463fff700 1 -- 172.21.15.113:0/3020710518 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646800ba00 con 0x7f646c06c470 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.155+0000 7f6463fff700 1 -- 172.21.15.113:0/3020710518 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800bd90 con 0x7f646c06c470 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 -- 172.21.15.113:0/3020710518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c06c470 msgr2=0x7f646c0b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 --2- 172.21.15.113:0/3020710518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c06c470 0x7f646c0b3d90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f64680098f0 tx=0x7f6468009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 -- 172.21.15.113:0/3020710518 shutdown_connections 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 --2- 172.21.15.113:0/3020710518 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c0b42d0 0x7f646c06dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 --2- 172.21.15.113:0/3020710518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c06c470 0x7f646c0b3d90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 -- 172.21.15.113:0/3020710518 >> 172.21.15.113:0/3020710518 conn(0x7f646c0b3130 msgr2=0x7f646c0d0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 -- 172.21.15.113:0/3020710518 shutdown_connections 2024-01-16T13:53:09.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.156+0000 7f6473931700 1 -- 172.21.15.113:0/3020710518 wait complete. 2024-01-16T13:53:09.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.157+0000 7f6473931700 1 Processor -- start 2024-01-16T13:53:09.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.157+0000 7f6473931700 1 -- start start 2024-01-16T13:53:09.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f646c06c470 0x7f646c1f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c073f90 0x7f646c133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 0x7f646c133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c1f3800 con 0x7f646c0b42d0 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c0c5da0 con 0x7f646c06c470 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c134e70 con 0x7f646c073f90 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 0x7f646c133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 0x7f646c133bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50070/0 (socket says 172.21.15.113:50070) 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 -- 172.21.15.113:0/2358883993 learned_addr learned my addr 172.21.15.113:0/2358883993 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f64716cd700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f646c06c470 0x7f646c1f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 -- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c073f90 msgr2=0x7f646c133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6470ecc700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c073f90 0x7f646c133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c073f90 0x7f646c133680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 -- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f646c06c470 msgr2=0x7f646c1f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f646c06c470 0x7f646c1f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 -- 172.21.15.113:0/2358883993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468009550 con 0x7f646c0b42d0 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f64716cd700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f646c06c470 0x7f646c1f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6471ece700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 0x7f646c133bc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f646400b970 tx=0x7f646400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646400c9b0 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646400cb30 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c134260 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.158+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464014440 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.159+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c134820 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.160+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c067f60 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.160+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f646401d3b0 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.160+0000 7f6461ffb700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f645006b350 0x7f645006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.160+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6464006e40 con 0x7f646c0b42d0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.160+0000 7f64716cd700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f645006b350 0x7f645006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.161+0000 7f64716cd700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f645006b350 0x7f645006d810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f645c000f60 tx=0x7f645c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.165+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64640547f0 con 0x7f646c0b42d0 2024-01-16T13:53:09.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.394+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f646c134fb0 con 0x7f646c0b42d0 2024-01-16T13:53:09.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.395+0000 7f6461ffb700 1 -- 172.21.15.113:0/2358883993 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f6464019090 con 0x7f646c0b42d0 2024-01-16T13:53:09.377 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:09.377 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4915,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/2577054536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2577054536},{"type":"v1","addr":"172.21.15.124:6834","nonce":2577054536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5055,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6836/2172474833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6835","nonce":2172474833},{"type":"v1","addr":"172.21.15.113:6836","nonce":2172474833}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5058,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/2409334271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2409334271},{"type":"v1","addr":"172.21.15.113:6837","nonce":2409334271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T13:52:58.941819+0000","modified":"2024-01-16T13:53:01.960866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.124:6835/1874427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":1874427660},{"type":"v1","addr":"172.21.15.124:6835","nonce":1874427660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T13:53:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f645006b350 msgr2=0x7f645006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f645006b350 0x7f645006d810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f645c000f60 tx=0x7f645c009040 comp rx=0 tx=0).stop 2024-01-16T13:53:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 msgr2=0x7f646c133bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 0x7f646c133bc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f646400b970 tx=0x7f646400bd30 comp rx=0 tx=0).stop 2024-01-16T13:53:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 shutdown_connections 2024-01-16T13:53:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f645006b350 0x7f645006d810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f646c0b42d0 0x7f646c133bc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f646c06c470 0x7f646c1f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 --2- 172.21.15.113:0/2358883993 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f646c073f90 0x7f646c133680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 >> 172.21.15.113:0/2358883993 conn(0x7f646c0b3130 msgr2=0x7f646c06e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 shutdown_connections 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.397+0000 7f6473931700 1 -- 172.21.15.113:0/2358883993 wait complete. 2024-01-16T13:53:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 16 2024-01-16T13:53:09.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T13:52:58.941819+0000', 'modified': '2024-01-16T13:53:01.960866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5067}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5067': {'gid': 5067, 'name': 'd', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.124:6835/1874427660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.124:6833', 'nonce': 1874427660}, {'type': 'v1', 'addr': '172.21.15.124:6835', 'nonce': 1874427660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T13:53:09.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T13:53:09.795 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T13:53:09.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.937+0000 7feae8002700 1 Processor -- start 2024-01-16T13:53:09.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.938+0000 7feae8002700 1 -- start start 2024-01-16T13:53:09.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.938+0000 7feae8002700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7feae006ba00 0x7feae00ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.938+0000 7feae8002700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae00cff40 con 0x7feae0073f90 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.938+0000 7feae8002700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae006be90 con 0x7feae0162590 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.938+0000 7feae8002700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae00ce760 con 0x7feae006ba00 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae659f700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7feae0073f90 0x7feae0074370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:45264/0 (socket says 172.21.15.113:45264) 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae659f700 1 -- 172.21.15.113:0/3219394277 learned_addr learned my addr 172.21.15.113:0/3219394277 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864421850 0 0) 0x7feae00cff40 con 0x7feae0073f90 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae559d700 1 --2- 172.21.15.113:0/3219394277 >> v2:172.21.15.124:3300/0 conn(0x7feae006ba00 0x7feae00ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feacc0035b0 con 0x7feae0073f90 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186624716 0 0) 0x7feacc0035b0 con 0x7feae0073f90 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 >> v2:172.21.15.124:3300/0 conn(0x7feae006ba00 msgr2=0x7feae00ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 --2- 172.21.15.113:0/3219394277 >> v2:172.21.15.124:3300/0 conn(0x7feae006ba00 0x7feae00ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 >> v1:172.21.15.124:6789/0 conn(0x7feae0162590 legacy=0x7feae00d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae006c100 con 0x7feae0073f90 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fead80019f0 con 0x7feae0073f90 2024-01-16T13:53:09.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 >> v1:172.21.15.113:6789/0 conn(0x7feae0073f90 legacy=0x7feae0074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feacc003a00 0x7feacc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feacc004850 0x7feacc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc005210 0x7feacc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00ce760 con 0x7feacc003a00 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc00c160 con 0x7feacc005210 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc00c320 con 0x7feacc004850 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae559d700 1 --2- 172.21.15.113:0/3219394277 >> v2:172.21.15.124:3300/0 conn(0x7feae006ba00 0x7feae00ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feacc003a00 0x7feacc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae659f700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc005210 0x7feacc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.939+0000 7feae559d700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feacc004850 0x7feacc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae5d9e700 1 -- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feacc004850 msgr2=0x7feacc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feacc004850 0x7feacc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae5d9e700 1 -- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc005210 msgr2=0x7feacc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc005210 0x7feacc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae5d9e700 1 -- 172.21.15.113:0/3219394277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc0035b0 con 0x7feacc003a00 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae659f700 1 -- 172.21.15.113:0/3219394277 reap_dead start 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feacc003a00 0x7feacc003e20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7feadc0017c0 tx=0x7feacc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b21ac1b12cef4e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc0052d0 con 0x7feacc003a00 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc005450 con 0x7feacc003a00 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae4d9c700 1 -- 172.21.15.113:0/3219394277 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc0038c0 con 0x7feacc003a00 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 -- 172.21.15.113:0/3219394277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feacc003a00 msgr2=0x7feacc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feacc003a00 0x7feacc003e20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7feadc0017c0 tx=0x7feacc005880 comp rx=0 tx=0).stop 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 -- 172.21.15.113:0/3219394277 shutdown_connections 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 --2- 172.21.15.113:0/3219394277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feacc003a00 0x7feacc003e20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 -- 172.21.15.113:0/3219394277 >> 172.21.15.113:0/3219394277 conn(0x7feae00cd570 msgr2=0x7feae00cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 -- 172.21.15.113:0/3219394277 shutdown_connections 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.940+0000 7feae8002700 1 -- 172.21.15.113:0/3219394277 wait complete. 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 Processor -- start 2024-01-16T13:53:09.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 -- start start 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc003a00 0x7feae014c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feae0166930 0x7feae0157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 0x7feae0162c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae01587f0 con 0x7feae0158370 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae014e880 con 0x7feacc003a00 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae8002700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae014d400 con 0x7feae0166930 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae659f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 0x7feae0162c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae659f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 0x7feae0162c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50094/0 (socket says 172.21.15.113:50094) 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.942+0000 7feae659f700 1 -- 172.21.15.113:0/3079964869 learned_addr learned my addr 172.21.15.113:0/3079964869 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc003a00 0x7feae014c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae559d700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feae0166930 0x7feae0157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae659f700 1 -- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feae0166930 msgr2=0x7feae0157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae659f700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feae0166930 0x7feae0157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae659f700 1 -- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc003a00 msgr2=0x7feae014c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae659f700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc003a00 0x7feae014c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae659f700 1 -- 172.21.15.113:0/3079964869 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae014d6e0 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae659f700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 0x7feae0162c70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feae006d090 tx=0x7fead801b5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7fead67fc700 1 -- 172.21.15.113:0/3079964869 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead801be00 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7fead67fc700 1 -- 172.21.15.113:0/3079964869 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80243f0 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae014d9c0 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7fead67fc700 1 -- 172.21.15.113:0/3079964869 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8024780 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.943+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae014df80 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.945+0000 7fead67fc700 1 -- 172.21.15.113:0/3079964869 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fead80248e0 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.945+0000 7fead67fc700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feac406b350 0x7feac406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.945+0000 7fead67fc700 1 -- 172.21.15.113:0/3079964869 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fead809b460 con 0x7feae0158370 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.945+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feac406b350 0x7feac406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.945+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feac406b350 0x7feac406d810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feadc000c00 tx=0x7feadc009030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.947+0000 7feaf41feb80 1 Processor -- start 2024-01-16T13:53:09.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.947+0000 7feaf41feb80 1 -- start start 2024-01-16T13:53:09.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 0x55702c152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c12f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c109dd0 0x55702c10c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feaf41feb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702c07f480 con 0x55702c097ce0 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702c065770 con 0x55702c109dd0 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55702c063090 con 0x55702c134410 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 0x55702c152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 0x55702c152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50098/0 (socket says 172.21.15.113:50098) 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 -- 172.21.15.113:0/663925391 learned_addr learned my addr 172.21.15.113:0/663925391 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae659f700 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c109dd0 0x55702c10c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae559d700 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c12f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 -- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 msgr2=0x55702c12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c12f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 -- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c109dd0 msgr2=0x55702c10c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c109dd0 0x55702c10c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 -- 172.21.15.113:0/663925391 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55702be87b10 con 0x55702c097ce0 2024-01-16T13:53:09.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feae5d9e700 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 0x55702c152cb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feadc005040 tx=0x7feadc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e781679f23721090 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.948+0000 7feac37fe700 1 -- 172.21.15.113:0/663925391 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc003680 con 0x55702c097ce0 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feac37fe700 1 -- 172.21.15.113:0/663925391 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc0037e0 con 0x55702c097ce0 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feac37fe700 1 -- 172.21.15.113:0/663925391 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc003b70 con 0x55702c097ce0 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 -- 172.21.15.113:0/663925391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 msgr2=0x55702c152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 0x55702c152cb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feadc005040 tx=0x7feadc011000 comp rx=0 tx=0).stop 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 -- 172.21.15.113:0/663925391 shutdown_connections 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c109dd0 0x55702c10c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c097ce0 0x55702c152cb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 --2- 172.21.15.113:0/663925391 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c12f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 -- 172.21.15.113:0/663925391 >> 172.21.15.113:0/663925391 conn(0x55702c039410 msgr2=0x55702c151830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 -- 172.21.15.113:0/663925391 shutdown_connections 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.949+0000 7feaf41feb80 1 -- 172.21.15.113:0/663925391 wait complete. 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.950+0000 7feaf41feb80 1 -- start start 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.950+0000 7feaf41feb80 1 Processor -- start 2024-01-16T13:53:09.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c12be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c148e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702c05e980 con 0x55702c109dd0 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702bffe000 con 0x55702c097ce0 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55702bffdca0 con 0x55702c134410 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feaf41feb80 10 client.0 _ll_register_callbacks cb 0x55702be61350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50110/0 (socket says 172.21.15.113:50110) 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 -- 172.21.15.113:0/3057941970 learned_addr learned my addr 172.21.15.113:0/3057941970 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae659f700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c12be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 -- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 msgr2=0x55702c148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 -- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 msgr2=0x55702c12be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c12be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 -- 172.21.15.113:0/3057941970 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc0035b0 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feae559d700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12c350 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fead00132e0 tx=0x7fead00135f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356a50ec582523f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.951+0000 7feac0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feac0ff9700 1 -- 172.21.15.113:0/3057941970 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0013e80 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feac0ff9700 1 -- 172.21.15.113:0/3057941970 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fead0016400 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55702c07e450 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feac0ff9700 1 -- 172.21.15.113:0/3057941970 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fead00165c0 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 10 client.5274 fetch_fsmap learned FSMap version 16 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55702c020290 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feac0ff9700 1 -- 172.21.15.113:0/3057941970 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55702c07e450 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feac0ff9700 1 -- 172.21.15.113:0/3057941970 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fead0016720 con 0x55702c109dd0 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 10 client.5274 fetch_fsmap finished waiting for FSMap version 16 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 10 client.5274 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 20 client.5274 populate_metadata read hostname 'smithi113' 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c13a010 0x55702c152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 4 client.5274 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T13:53:09.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 --> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55702c057960 con 0x55702c13a010 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.952+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c13a010 0x55702c152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.953+0000 7feac0ff9700 10 client.5274 ms_handle_connect on v2:172.21.15.124:6833/1874427660 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.953+0000 7feae5d9e700 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c13a010 0x55702c152cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de6c1efd9f3985a server_cookie=3186c90e49d93fcb in_seq=0 out_seq=0 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.953+0000 7feac0ff9700 1 -- 172.21.15.113:0/3057941970 <== mds.0 v2:172.21.15.124:6833/1874427660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55702c057960 con 0x55702c13a010 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.953+0000 7feac0ff9700 10 client.5274 handle_command_reply: tid=0 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.953+0000 7feaf41feb80 1 client.5274 shutdown 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 msgr2=0x55702c12c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12c350 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fead00132e0 tx=0x7fead00135f0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 shutdown_connections 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c134410 0x55702c148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c12be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c13a010 0x55702c152cb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 --2- 172.21.15.113:0/3057941970 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12c350 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feac0ff9700 0 client.5274 ms_handle_reset on v2:172.21.15.124:6833/1874427660 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 >> 172.21.15.113:0/3057941970 conn(0x55702c039410 msgr2=0x55702c10b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 shutdown_connections 2024-01-16T13:53:09.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 1 -- 172.21.15.113:0/3057941970 wait complete. 2024-01-16T13:53:09.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.954+0000 7feaf41feb80 20 client.5274 trim_cache size 0 max 16384 2024-01-16T13:53:09.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.972+0000 7feaf41feb80 1 Processor -- start 2024-01-16T13:53:09.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.972+0000 7feaf41feb80 1 -- start start 2024-01-16T13:53:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c097ce0 0x55702c106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c134410 0x55702c12e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feaf41feb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702c13a6c0 con 0x55702c109dd0 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702bedd650 con 0x55702c134410 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55702c12b910 con 0x55702c097ce0 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50116/0 (socket says 172.21.15.113:50116) 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 -- 172.21.15.113:0/2590568319 learned_addr learned my addr 172.21.15.113:0/2590568319 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 -- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c097ce0 msgr2=0x55702c106660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c097ce0 0x55702c106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae659f700 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c134410 0x55702c12e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 -- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c134410 msgr2=0x55702c12e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c134410 0x55702c12e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae559d700 1 -- 172.21.15.113:0/2590568319 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0163af0 con 0x55702c109dd0 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.973+0000 7feae659f700 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c134410 0x55702c12e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:09.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feae559d700 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12e3c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fead0006d00 tx=0x7fead0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3c9685288b4bca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feac3fff700 1 -- 172.21.15.113:0/2590568319 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0013e80 con 0x55702c109dd0 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feac3fff700 1 -- 172.21.15.113:0/2590568319 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead000c040 con 0x55702c109dd0 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feac3fff700 1 -- 172.21.15.113:0/2590568319 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0016400 con 0x55702c109dd0 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 -- 172.21.15.113:0/2590568319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 msgr2=0x55702c12e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12e3c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fead0006d00 tx=0x7fead0018000 comp rx=0 tx=0).stop 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 -- 172.21.15.113:0/2590568319 shutdown_connections 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c134410 0x55702c12e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c097ce0 0x55702c106660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2590568319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c109dd0 0x55702c12e3c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 -- 172.21.15.113:0/2590568319 >> 172.21.15.113:0/2590568319 conn(0x55702c039410 msgr2=0x55702c133730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 -- 172.21.15.113:0/2590568319 shutdown_connections 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.974+0000 7feaf41feb80 1 -- 172.21.15.113:0/2590568319 wait complete. 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.975+0000 7feaf41feb80 1 -- start start 2024-01-16T13:53:09.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.975+0000 7feaf41feb80 1 Processor -- start 2024-01-16T13:53:09.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c109dd0 0x55702c12aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 0x55702c12dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702c1a1630 con 0x55702c134410 2024-01-16T13:53:09.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55702c107ef0 con 0x55702c097ce0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55702c12aa40 con 0x55702c109dd0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 10 client.0 _ll_register_callbacks cb 0x55702be61350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 0x55702c12dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 0x55702c12dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50120/0 (socket says 172.21.15.113:50120) 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 -- 172.21.15.113:0/2269897437 learned_addr learned my addr 172.21.15.113:0/2269897437 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae5d9e700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae559d700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c109dd0 0x55702c12aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 -- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c109dd0 msgr2=0x55702c12aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c109dd0 0x55702c12aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 -- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 msgr2=0x55702c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 -- 172.21.15.113:0/2269897437 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55702c020290 con 0x55702c134410 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae5d9e700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feae659f700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 0x55702c12dfa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fead8071d60 tx=0x7fead806a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4884fb7d05e329 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:09.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feac0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feac0ff9700 1 -- 172.21.15.113:0/2269897437 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead809b0b0 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feac0ff9700 1 -- 172.21.15.113:0/2269897437 <== mon.0 v2:172.21.15.113:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fead809b210 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.976+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feadc0037e0 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feac0ff9700 1 -- 172.21.15.113:0/2269897437 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fead806a1f0 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 10 client.5280 fetch_fsmap learned FSMap version 16 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55702bedfc80 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feac0ff9700 1 -- 172.21.15.113:0/2269897437 <== mon.0 v2:172.21.15.113:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc0037e0 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feac0ff9700 1 -- 172.21.15.113:0/2269897437 <== mon.0 v2:172.21.15.113:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fead8069480 con 0x55702c134410 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 10 client.5280 fetch_fsmap finished waiting for FSMap version 16 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 10 client.5280 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 20 client.5280 populate_metadata read hostname 'smithi113' 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c10b740 0x55702c12e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 4 client.5280 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 --> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55702c057680 con 0x55702c10b740 2024-01-16T13:53:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feae5d9e700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c10b740 0x55702c12e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feae5d9e700 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c10b740 0x55702c12e4e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a4d9a53d068fb82 server_cookie=d9e23602bd0a95da in_seq=0 out_seq=0 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.977+0000 7feac0ff9700 10 client.5280 ms_handle_connect on v2:172.21.15.124:6833/1874427660 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feac0ff9700 1 -- 172.21.15.113:0/2269897437 <== mds.0 v2:172.21.15.124:6833/1874427660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55702c057680 con 0x55702c10b740 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feac0ff9700 10 client.5280 handle_command_reply: tid=0 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 client.5280 shutdown 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 msgr2=0x55702c12dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 0x55702c12dfa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fead8071d60 tx=0x7fead806a7c0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 shutdown_connections 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:6833/1874427660,v1:172.21.15.124:6835/1874427660] conn(0x55702c10b740 0x55702c12e4e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55702c134410 0x55702c12dfa0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55702c097ce0 0x55702c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feac0ff9700 0 client.5280 ms_handle_reset on v2:172.21.15.124:6833/1874427660 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 --2- 172.21.15.113:0/2269897437 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55702c109dd0 0x55702c12aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 >> 172.21.15.113:0/2269897437 conn(0x55702c039410 msgr2=0x55702c148c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 shutdown_connections 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 1 -- 172.21.15.113:0/2269897437 wait complete. 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feaf41feb80 20 client.5280 trim_cache size 0 max 16384 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "cluster_fsid": "e77ab909-7680-4e53-a08d-a26ae42f1a8c", 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "whoami": 0, 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 5067, 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "want_state": "up:active", 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "up:active", 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "fs_name": "cephfs", 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "rank_uptime": 10.435072142999999, 2024-01-16T13:53:09.966 INFO:teuthology.orchestra.run.smithi113.stdout: "mdsmap_epoch": 16, 2024-01-16T13:53:09.967 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch": 33, 2024-01-16T13:53:09.967 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_epoch_barrier": 33, 2024-01-16T13:53:09.967 INFO:teuthology.orchestra.run.smithi113.stdout: "uptime": 18.880328002999999 2024-01-16T13:53:09.967 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feac406b350 msgr2=0x7feac406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feac406b350 0x7feac406d810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feadc000c00 tx=0x7feadc009030 comp rx=0 tx=0).stop 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 msgr2=0x7feae0162c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 0x7feae0162c70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feae006d090 tx=0x7fead801b5f0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 shutdown_connections 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7feac406b350 0x7feac406d810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae0158370 0x7feae0162c70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7feacc003a00 0x7feae014c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 --2- 172.21.15.113:0/3079964869 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7feae0166930 0x7feae0157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.978+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 >> 172.21.15.113:0/3079964869 conn(0x7feae00cd570 msgr2=0x7feae00b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.979+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 shutdown_connections 2024-01-16T13:53:09.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:09.979+0000 7feae8002700 1 -- 172.21.15.113:0/3079964869 wait complete. 2024-01-16T13:53:09.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T13:53:10.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.112+0000 7f8436b5d700 1 Processor -- start 2024-01-16T13:53:10.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.112+0000 7f8436b5d700 1 -- start start 2024-01-16T13:53:10.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f8436b5d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f843006ee50 0x7f8430074370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:10.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f8436b5d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8430074e90 con 0x7f8430073f90 2024-01-16T13:53:10.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f8436b5d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84300748b0 con 0x7f843015c140 2024-01-16T13:53:10.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f8436b5d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843015c700 con 0x7f843006ee50 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f84348f9700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f8430073f90 0x7f84300ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:45268/0 (socket says 172.21.15.113:45268) 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f84348f9700 1 -- 172.21.15.113:0/2796557049 learned_addr learned my addr 172.21.15.113:0/2796557049 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842ffff700 1 --2- 172.21.15.113:0/2796557049 >> v2:172.21.15.124:3300/0 conn(0x7f843006ee50 0x7f8430074370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842f7fe700 1 -- 172.21.15.113:0/2796557049 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708170312 0 0) 0x7f8430074e90 con 0x7f8430073f90 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842f7fe700 1 -- 172.21.15.113:0/2796557049 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84200035b0 con 0x7f8430073f90 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842ffff700 1 -- 172.21.15.113:0/2796557049 >> v1:172.21.15.124:6789/0 conn(0x7f843015c140 legacy=0x7f84300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842f7fe700 1 -- 172.21.15.113:0/2796557049 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961094451 0 0) 0x7f84300748b0 con 0x7f843015c140 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842ffff700 1 -- 172.21.15.113:0/2796557049 >> v1:172.21.15.113:6789/0 conn(0x7f8430073f90 legacy=0x7f84300ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842ffff700 1 -- 172.21.15.113:0/2796557049 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843015c9e0 con 0x7f843006ee50 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.113+0000 7f842ffff700 1 --2- 172.21.15.113:0/2796557049 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f843006ee50 0x7f8430074370 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f841c009870 tx=0x7f841c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9201c64d1fe78c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f842f7fe700 1 -- 172.21.15.113:0/2796557049 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84240015f0 con 0x7f843006ee50 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f842f7fe700 1 -- 172.21.15.113:0/2796557049 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c004030 con 0x7f843006ee50 2024-01-16T13:53:10.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f842f7fe700 1 -- 172.21.15.113:0/2796557049 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c0043c0 con 0x7f843006ee50 2024-01-16T13:53:10.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 -- 172.21.15.113:0/2796557049 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f843006ee50 msgr2=0x7f8430074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2796557049 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f843006ee50 0x7f8430074370 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f841c009870 tx=0x7f841c009b80 comp rx=0 tx=0).stop 2024-01-16T13:53:10.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 -- 172.21.15.113:0/2796557049 shutdown_connections 2024-01-16T13:53:10.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2796557049 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f843006ee50 0x7f8430074370 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 -- 172.21.15.113:0/2796557049 >> 172.21.15.113:0/2796557049 conn(0x7f84300cd570 msgr2=0x7f84300cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:10.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 -- 172.21.15.113:0/2796557049 shutdown_connections 2024-01-16T13:53:10.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.114+0000 7f8436b5d700 1 -- 172.21.15.113:0/2796557049 wait complete. 2024-01-16T13:53:10.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.115+0000 7f8436b5d700 1 Processor -- start 2024-01-16T13:53:10.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.115+0000 7f8436b5d700 1 -- start start 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f8436b5d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f843006ee50 0x7f8430148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f8436b5d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8430073f90 0x7f84301577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f8436b5d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 0x7f8430157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f8436b5d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843015a590 con 0x7f843015c140 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f8436b5d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843015a870 con 0x7f8430073f90 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f8436b5d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843015ab50 con 0x7f843006ee50 2024-01-16T13:53:10.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f84350fa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 0x7f8430157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f84350fa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 0x7f8430157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50130/0 (socket says 172.21.15.113:50130) 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.116+0000 7f84350fa700 1 -- 172.21.15.113:0/2427063263 learned_addr learned my addr 172.21.15.113:0/2427063263 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84348f9700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f843006ee50 0x7f8430148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f842ffff700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8430073f90 0x7f84301577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84350fa700 1 -- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f843006ee50 msgr2=0x7f8430148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84350fa700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f843006ee50 0x7f8430148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84350fa700 1 -- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8430073f90 msgr2=0x7f84301577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84350fa700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8430073f90 0x7f84301577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84350fa700 1 -- 172.21.15.113:0/2427063263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c0094a0 con 0x7f843015c140 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84348f9700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f843006ee50 0x7f8430148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f842ffff700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8430073f90 0x7f84301577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f84350fa700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 0x7f8430157d10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f842800c7f0 tx=0x7f842800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428003e30 con 0x7f843015c140 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8428005070 con 0x7f843015c140 2024-01-16T13:53:10.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843015ae30 con 0x7f843015c140 2024-01-16T13:53:10.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84280053e0 con 0x7f843015c140 2024-01-16T13:53:10.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.117+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843015a320 con 0x7f843015c140 2024-01-16T13:53:10.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.118+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84300623b0 con 0x7f843015c140 2024-01-16T13:53:10.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.118+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8428005540 con 0x7f843015c140 2024-01-16T13:53:10.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.119+0000 7f842d7fa700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f841406b350 0x7f841406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:10.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.119+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8428089860 con 0x7f843015c140 2024-01-16T13:53:10.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.119+0000 7f84348f9700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f841406b350 0x7f841406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:10.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.119+0000 7f84348f9700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f841406b350 0x7f841406d810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84240017e0 tx=0x7f84240016f0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:10.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.123+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8428054760 con 0x7f843015c140 2024-01-16T13:53:10.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.357+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f843015b020 con 0x7f843015c140 2024-01-16T13:53:10.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.358+0000 7f842d7fa700 1 -- 172.21.15.113:0/2427063263 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8428057d80 con 0x7f843015c140 2024-01-16T13:53:10.339 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T13:53:10.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f841406b350 msgr2=0x7f841406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f841406b350 0x7f841406d810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84240017e0 tx=0x7f84240016f0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 msgr2=0x7f8430157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:10.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 0x7f8430157d10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f842800c7f0 tx=0x7f842800cbb0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 shutdown_connections 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f841406b350 0x7f841406d810 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f843015c140 0x7f8430157d10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f843006ee50 0x7f8430148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 --2- 172.21.15.113:0/2427063263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8430073f90 0x7f84301577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 >> 172.21.15.113:0/2427063263 conn(0x7f84300cd570 msgr2=0x7f84300cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 shutdown_connections 2024-01-16T13:53:10.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:10.360+0000 7f8436b5d700 1 -- 172.21.15.113:0/2427063263 wait complete. 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T13:53:10.354 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T13:53:10.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.355 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-01-16T13:53:10.372 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:53:10.372 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:53:10.372 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:10.373 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42435sec preferred_lft 42435sec 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::9d38:c2c1:4a69:a61c/64 scope link noprefixroute 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 62:af:26:5d:98:8b brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::866:67ff:fe19:1e59/64 scope link 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 62:af:26:5d:98:8b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::60af:26ff:fe5d:988b/64 scope link 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.374 INFO:teuthology.orchestra.run.smithi113.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T13:53:10.375 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 8e:25:11:e7:fd:b7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T13:53:10.375 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::8c25:11ff:fee7:fdb7/64 scope link 2024-01-16T13:53:10.375 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:10.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.375 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T13:53:10.438 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-01-16T13:53:10.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.439 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-01-16T13:53:10.498 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-01-16T13:53:10.498 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T13:53:10.498 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T13:53:10.498 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T13:53:10.498 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T13:53:10.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.498 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-01-16T13:53:10.499 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-01-16T13:53:10.499 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T13:53:10.499 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T13:53:10.499 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T13:53:10.499 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-01-16T13:53:10.527 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:53:10.541 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:53:10.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.604 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-01-16T13:53:10.662 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T13:53:10.662 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T13:53:10.662 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T13:53:10.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.0 up 2024-01-16T13:53:10.729 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:53:10.729 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T13:53:10.785 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T13:53:10.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:53:10.785 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T13:53:10.842 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-16T13:53:10.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.842 DEBUG:teuthology.orchestra.run.smithi113:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:53:10.928 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:53:10.928 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-16T13:53:10.928 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:53:10.937 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:53:10.937 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-16T13:53:10.937 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:53:10.937 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:53:10.937 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:53:10.956 INFO:teuthology.orchestra.run.smithi113.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:invalid new device string format 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790" 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2024-01-16T13:53:10.957 INFO:teuthology.orchestra.run.smithi113.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e77ab909-7680-4e53-a08d-a26ae42f1a8c" will pass to kernel 2024-01-16T13:53:10.959 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-16T13:53:10.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.959 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T13:53:10.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:10.986 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:53:11.074 INFO:teuthology.orchestra.run.smithi113.stdout:client5041 2024-01-16T13:53:11.074 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:53:11.074 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:11.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T13:53:11.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.269+0000 7fa9fc6fd700 1 Processor -- start 2024-01-16T13:53:11.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.270+0000 7fa9fc6fd700 1 -- start start 2024-01-16T13:53:11.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.270+0000 7fa9fc6fd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f415e2a0 0x7fa9f406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:11.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9fc6fd700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa9f40ce090 0x7fa9f406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:11.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9fc6fd700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f406e7f0 con 0x7fa9f4073f90 2024-01-16T13:53:11.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9fc6fd700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f406dec0 con 0x7fa9f40ce090 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9fc6fd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f406e1a0 con 0x7fa9f415e2a0 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa9f40ce090 0x7fa9f406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa9f40ce090 0x7fa9f406ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50144/0 (socket says 172.21.15.113:50144) 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 -- 172.21.15.113:0/2396528 learned_addr learned my addr 172.21.15.113:0/2396528 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9fa499700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f415e2a0 0x7fa9f406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 -- 172.21.15.113:0/2396528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f415e2a0 msgr2=0x7fa9f406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f415e2a0 0x7fa9f406ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 -- 172.21.15.113:0/2396528 >> v1:172.21.15.124:6789/0 conn(0x7fa9f4073f90 legacy=0x7fa9f40cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 -- 172.21.15.113:0/2396528 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f406e480 con 0x7fa9f40ce090 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9fa499700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f415e2a0 0x7fa9f406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9c98700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f406ec80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00098f0 tx=0x7fa9f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52002d2fcb2dfa96 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9497700 1 -- 172.21.15.113:0/2396528 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000b8a0 con 0x7fa9f40ce090 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9497700 1 -- 172.21.15.113:0/2396528 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f000ba00 con 0x7fa9f40ce090 2024-01-16T13:53:11.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.271+0000 7fa9f9497700 1 -- 172.21.15.113:0/2396528 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000bd90 con 0x7fa9f40ce090 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/2396528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 msgr2=0x7fa9f406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f406ec80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00098f0 tx=0x7fa9f0009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/2396528 shutdown_connections 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f415e2a0 0x7fa9f406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/2396528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f406ec80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/2396528 >> 172.21.15.113:0/2396528 conn(0x7fa9f40cd570 msgr2=0x7fa9f40cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/2396528 shutdown_connections 2024-01-16T13:53:11.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.272+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/2396528 wait complete. 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.273+0000 7fa9fc6fd700 1 Processor -- start 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.273+0000 7fa9fc6fd700 1 -- start start 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fc6fd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f4073f90 0x7fa9f4159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fc6fd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f4146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fc6fd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f415e2a0 0x7fa9f4071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fc6fd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f4158d00 con 0x7fa9f40ce090 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fc6fd700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f4156fc0 con 0x7fa9f415e2a0 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fc6fd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f41583c0 con 0x7fa9f4073f90 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f4146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:11.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f4146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50160/0 (socket says 172.21.15.113:50160) 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 -- 172.21.15.113:0/3818134628 learned_addr learned my addr 172.21.15.113:0/3818134628 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fac9a700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f415e2a0 0x7fa9f4071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fa499700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f4073f90 0x7fa9f4159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 -- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f4073f90 msgr2=0x7fa9f4159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f4073f90 0x7fa9f4159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 -- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f415e2a0 msgr2=0x7fa9f4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f415e2a0 0x7fa9f4071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9f9c98700 1 -- 172.21.15.113:0/3818134628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0009550 con 0x7fa9f40ce090 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.274+0000 7fa9fac9a700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f415e2a0 0x7fa9f4071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.275+0000 7fa9f9c98700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f4146120 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009c40 tx=0x7fa9f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.275+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0004480 con 0x7fa9f40ce090 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.275+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f001f070 con 0x7fa9f40ce090 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.275+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00038a0 con 0x7fa9f40ce090 2024-01-16T13:53:11.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.275+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f4149570 con 0x7fa9f40ce090 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.275+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f4158920 con 0x7fa9f40ce090 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.276+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f4158e40 con 0x7fa9f40ce090 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.276+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa9f0021030 con 0x7fa9f40ce090 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.276+0000 7fa9e6ffd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa9d806b380 0x7fa9d806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.276+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9f008b240 con 0x7fa9f40ce090 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.277+0000 7fa9fa499700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa9d806b380 0x7fa9d806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:11.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.277+0000 7fa9fa499700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa9d806b380 0x7fa9d806d840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009990 tx=0x7fa9e8008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:11.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.281+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9f4158e40 con 0x7fa9f40ce090 2024-01-16T13:53:11.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.515+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa9f4158e40 con 0x7fa9f40ce090 2024-01-16T13:53:11.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.515+0000 7fa9e6ffd700 1 -- 172.21.15.113:0/3818134628 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa9f4158e40 con 0x7fa9f40ce090 2024-01-16T13:53:11.497 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T13:53:11.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa9d806b380 msgr2=0x7fa9d806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:11.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa9d806b380 0x7fa9d806d840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009990 tx=0x7fa9e8008040 comp rx=0 tx=0).stop 2024-01-16T13:53:11.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 msgr2=0x7fa9f4146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:11.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f4146120 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009c40 tx=0x7fa9f0004750 comp rx=0 tx=0).stop 2024-01-16T13:53:11.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 shutdown_connections 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa9d806b380 0x7fa9d806d840 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f415e2a0 0x7fa9f4071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9f4073f90 0x7fa9f4159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 --2- 172.21.15.113:0/3818134628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9f40ce090 0x7fa9f4146120 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 >> 172.21.15.113:0/3818134628 conn(0x7fa9f40cd570 msgr2=0x7fa9f40b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 shutdown_connections 2024-01-16T13:53:11.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:11.518+0000 7fa9fc6fd700 1 -- 172.21.15.113:0/3818134628 wait complete. 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-16T13:53:11.512 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-01-16T13:53:11.513 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T13:53:11.513 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T13:53:11.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:11.513 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:53:11.536 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42403sec preferred_lft 42403sec 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec99:5e7d:854c:8532/64 scope link noprefixroute 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0e:6d:52:8c:06:03 brd ff:ff:ff:ff:ff:ff 2024-01-16T13:53:11.537 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::6c76:17ff:fef4:cd32/64 scope link 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0e:6d:52:8c:06:03 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::c6d:52ff:fe8c:603/64 scope link 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether fa:2c:35:29:12:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::f82c:35ff:fe29:12ff/64 scope link 2024-01-16T13:53:11.538 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2024-01-16T13:53:11.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:11.539 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T13:53:11.607 INFO:teuthology.orchestra.run.smithi124.stdout:1 2024-01-16T13:53:11.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:11.609 DEBUG:teuthology.orchestra.run.smithi124:> route 2024-01-16T13:53:11.672 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2024-01-16T13:53:11.672 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T13:53:11.673 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T13:53:11.673 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T13:53:11.673 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T13:53:11.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:11.673 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2024-01-16T13:53:11.673 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2024-01-16T13:53:11.674 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T13:53:11.674 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T13:53:11.674 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T13:53:11.674 DEBUG:teuthology.orchestra.run.smithi124:> ') 2024-01-16T13:53:11.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:11.825 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2024-01-16T13:53:11.846 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T13:53:11.846 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T13:53:11.846 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T13:53:11.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:11.846 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2024-01-16T13:53:11.919 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:53:11.919 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T13:53:11.980 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T13:53:11.980 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T13:53:11.980 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T13:53:12.039 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-16T13:53:12.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:12.039 DEBUG:teuthology.orchestra.run.smithi124:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:53:12.134 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:53:12.135 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-16T13:53:12.135 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:53:12.144 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:53:12.144 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-16T13:53:12.145 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:53:12.145 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:53:12.145 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:3300,172.21.15.124:3300,172.21.15.124:3301" 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790" 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.113:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2024-01-16T13:53:12.164 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=e77ab909-7680-4e53-a08d-a26ae42f1a8c" will pass to kernel 2024-01-16T13:53:12.167 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-16T13:53:12.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:12.167 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T13:53:12.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:12.196 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-16T13:53:12.271 INFO:teuthology.orchestra.run.smithi124.stdout:client4799 2024-01-16T13:53:12.271 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2024-01-16T13:53:12.271 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:12.329 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T13:53:12.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.486+0000 7f76fc75f700 1 Processor -- start 2024-01-16T13:53:12.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.486+0000 7f76fc75f700 1 -- start start 2024-01-16T13:53:12.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fc75f700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76f40b01e0 0x7f76f406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fc75f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f76f40b59c0 0x7f76f40b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fc75f700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f40b05c0 con 0x7f76f4073f90 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fc75f700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f41470a0 con 0x7f76f40b01e0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fc75f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f40b5e70 con 0x7f76f40b59c0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76f40b01e0 0x7f76f406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f76f40b01e0 0x7f76f406e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50176/0 (socket says 172.21.15.113:50176) 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 -- 172.21.15.113:0/472616761 learned_addr learned my addr 172.21.15.113:0/472616761 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76f9cfa700 1 --2- 172.21.15.113:0/472616761 >> v2:172.21.15.124:3300/0 conn(0x7f76f40b59c0 0x7f76f40b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 -- 172.21.15.113:0/472616761 >> v2:172.21.15.124:3300/0 conn(0x7f76f40b59c0 msgr2=0x7f76f40b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 --2- 172.21.15.113:0/472616761 >> v2:172.21.15.124:3300/0 conn(0x7f76f40b59c0 0x7f76f40b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 -- 172.21.15.113:0/472616761 >> v1:172.21.15.124:6789/0 conn(0x7f76f4073f90 legacy=0x7f76f406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.487+0000 7f76fa4fb700 1 -- 172.21.15.113:0/472616761 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f40b4250 con 0x7f76f40b01e0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fa4fb700 1 --2- 172.21.15.113:0/472616761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b01e0 0x7f76f406e9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f76f4074fb0 tx=0x7f76e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c1b2c37e7b1153 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76f94f9700 1 -- 172.21.15.113:0/472616761 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4004230 con 0x7f76f40b01e0 2024-01-16T13:53:12.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76f94f9700 1 -- 172.21.15.113:0/472616761 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e4004390 con 0x7f76f40b01e0 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76f94f9700 1 -- 172.21.15.113:0/472616761 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e40036a0 con 0x7f76f40b01e0 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 -- 172.21.15.113:0/472616761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b01e0 msgr2=0x7f76f406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 --2- 172.21.15.113:0/472616761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b01e0 0x7f76f406e9c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f76f4074fb0 tx=0x7f76e4009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 -- 172.21.15.113:0/472616761 shutdown_connections 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 --2- 172.21.15.113:0/472616761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b01e0 0x7f76f406e9c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 --2- 172.21.15.113:0/472616761 >> v2:172.21.15.124:3300/0 conn(0x7f76f40b59c0 0x7f76f40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 -- 172.21.15.113:0/472616761 >> 172.21.15.113:0/472616761 conn(0x7f76f40d0b40 msgr2=0x7f76f40d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.488+0000 7f76fc75f700 1 -- 172.21.15.113:0/472616761 shutdown_connections 2024-01-16T13:53:12.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.489+0000 7f76fc75f700 1 -- 172.21.15.113:0/472616761 wait complete. 2024-01-16T13:53:12.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.490+0000 7f76fc75f700 1 Processor -- start 2024-01-16T13:53:12.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.490+0000 7f76fc75f700 1 -- start start 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.490+0000 7f76fc75f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f76f4073f90 0x7f76f413b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.490+0000 7f76fc75f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76f40b01e0 0x7f76f413cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.490+0000 7f76fc75f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 0x7f76f413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76fc75f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4161160 con 0x7f76f40b59c0 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76fc75f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4160550 con 0x7f76f40b01e0 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76fc75f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4160830 con 0x7f76f4073f90 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 0x7f76f413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 0x7f76f413f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50180/0 (socket says 172.21.15.113:50180) 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 -- 172.21.15.113:0/2560125657 learned_addr learned my addr 172.21.15.113:0/2560125657 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76f9cfa700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76f40b01e0 0x7f76f413cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76fa4fb700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f76f4073f90 0x7f76f413b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 -- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f76f4073f90 msgr2=0x7f76f413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f76f4073f90 0x7f76f413b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 -- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76f40b01e0 msgr2=0x7f76f413cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:12.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76f40b01e0 0x7f76f413cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 -- 172.21.15.113:0/2560125657 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4009550 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76facfc700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 0x7f76f413f5e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00cc10 tx=0x7f76ec00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec00aa60 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec00abc0 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f4160b10 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec012420 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.491+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f413b920 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.492+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f41612d0 con 0x7f76f40b59c0 2024-01-16T13:53:12.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.493+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f76ec012580 con 0x7f76f40b59c0 2024-01-16T13:53:12.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.493+0000 7f76eaffd700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f76d806b350 0x7f76d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:12.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.493+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76ec089ea0 con 0x7f76f40b59c0 2024-01-16T13:53:12.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.493+0000 7f76fa4fb700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f76d806b350 0x7f76d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:12.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.494+0000 7f76fa4fb700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f76d806b350 0x7f76d806d810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009130 tx=0x7f76e4008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:12.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.498+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76ec054d10 con 0x7f76f40b59c0 2024-01-16T13:53:12.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:12.733+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f76f41614f0 con 0x7f76f40b59c0 2024-01-16T13:53:13.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.601+0000 7f76eaffd700 1 -- 172.21.15.113:0/2560125657 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 136+0+0 (secure 0 0 0) 0x7f76ec058330 con 0x7f76f40b59c0 2024-01-16T13:53:13.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f76d806b350 msgr2=0x7f76d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f76d806b350 0x7f76d806d810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009130 tx=0x7f76e4008030 comp rx=0 tx=0).stop 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 msgr2=0x7f76f413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 0x7f76f413f5e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00cc10 tx=0x7f76ec00cf20 comp rx=0 tx=0).stop 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 shutdown_connections 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f76d806b350 0x7f76d806d810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76f40b59c0 0x7f76f413f5e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f76f4073f90 0x7f76f413b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 --2- 172.21.15.113:0/2560125657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76f40b01e0 0x7f76f413cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 >> 172.21.15.113:0/2560125657 conn(0x7f76f40d0b40 msgr2=0x7f76f4145950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 shutdown_connections 2024-01-16T13:53:13.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.604+0000 7f76fc75f700 1 -- 172.21.15.113:0/2560125657 wait complete. 2024-01-16T13:53:13.601 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T13:53:13.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.745+0000 7fa460fa1700 1 Processor -- start 2024-01-16T13:53:13.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.745+0000 7fa460fa1700 1 -- start start 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.746+0000 7fa460fa1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c15e1e0 0x7fa45c0d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.746+0000 7fa460fa1700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa45c0d0f80 0x7fa45c0b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.746+0000 7fa460fa1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45c0d20b0 con 0x7fa45c0d3a00 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.746+0000 7fa460fa1700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45c0d3150 con 0x7fa45c0d0f80 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.746+0000 7fa460fa1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c15e5c0 con 0x7fa45c15e1e0 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.746+0000 7fa45ad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa45c0d0f80 0x7fa45c0b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa45c0d0f80 0x7fa45c0b6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59852/0 (socket says 172.21.15.113:59852) 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 -- 172.21.15.113:0/2125310394 learned_addr learned my addr 172.21.15.113:0/2125310394 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45a59c700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c15e1e0 0x7fa45c0d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 -- 172.21.15.113:0/2125310394 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c15e1e0 msgr2=0x7fa45c0d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c15e1e0 0x7fa45c0d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 -- 172.21.15.113:0/2125310394 >> v1:172.21.15.124:6789/0 conn(0x7fa45c0d3a00 legacy=0x7fa45c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 -- 172.21.15.113:0/2125310394 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c0d2390 con 0x7fa45c0d0f80 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45a59c700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c15e1e0 0x7fa45c0d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45ad9d700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0b6ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa454009e80 tx=0x7fa45400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4f32a16bd0b8a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45959a700 1 -- 172.21.15.113:0/2125310394 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45400e070 con 0x7fa45c0d0f80 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45959a700 1 -- 172.21.15.113:0/2125310394 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45400b980 con 0x7fa45c0d0f80 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.747+0000 7fa45959a700 1 -- 172.21.15.113:0/2125310394 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45400bd30 con 0x7fa45c0d0f80 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 -- 172.21.15.113:0/2125310394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 msgr2=0x7fa45c0b6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0b6ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa454009e80 tx=0x7fa45400b330 comp rx=0 tx=0).stop 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 -- 172.21.15.113:0/2125310394 shutdown_connections 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0b6ef0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 --2- 172.21.15.113:0/2125310394 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c15e1e0 0x7fa45c0d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 -- 172.21.15.113:0/2125310394 >> 172.21.15.113:0/2125310394 conn(0x7fa45c0b5b50 msgr2=0x7fa45c0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 -- 172.21.15.113:0/2125310394 shutdown_connections 2024-01-16T13:53:13.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.748+0000 7fa460fa1700 1 -- 172.21.15.113:0/2125310394 wait complete. 2024-01-16T13:53:13.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.749+0000 7fa460fa1700 1 Processor -- start 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.749+0000 7fa460fa1700 1 -- start start 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c0d3a00 0x7fa45c1f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa45c15e1e0 0x7fa45c0ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c0cc290 con 0x7fa45c0d0f80 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c0cc570 con 0x7fa45c15e1e0 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c0cc850 con 0x7fa45c0d3a00 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59856/0 (socket says 172.21.15.113:59856) 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 -- 172.21.15.113:0/1623512770 learned_addr learned my addr 172.21.15.113:0/1623512770 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa459d9b700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c0d3a00 0x7fa45c1f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45ad9d700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa45c15e1e0 0x7fa45c0ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 -- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c0d3a00 msgr2=0x7fa45c1f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c0d3a00 0x7fa45c1f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 -- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa45c15e1e0 msgr2=0x7fa45c0ac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa45c15e1e0 0x7fa45c0ac140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 -- 172.21.15.113:0/1623512770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa454009b30 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45ad9d700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa45c15e1e0 0x7fa45c0ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa45a59c700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0cb940 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa45000e760 tx=0x7fa45000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450013610 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa450013770 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450013b20 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.750+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c0ccb30 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.751+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c0cc060 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.751+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c0751e0 con 0x7fa45c0d0f80 2024-01-16T13:53:13.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.752+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa450013c80 con 0x7fa45c0d0f80 2024-01-16T13:53:13.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.752+0000 7fa44affd700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa43c06b350 0x7fa43c06d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:13.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.752+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa450089720 con 0x7fa45c0d0f80 2024-01-16T13:53:13.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.752+0000 7fa459d9b700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa43c06b350 0x7fa43c06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:13.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.753+0000 7fa459d9b700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa43c06b350 0x7fa43c06d810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa45c1f6d10 tx=0x7fa44c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:13.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.757+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa450054620 con 0x7fa45c0d0f80 2024-01-16T13:53:13.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.974+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa45c0cc9c0 con 0x7fa43c06b350 2024-01-16T13:53:13.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.976+0000 7fa44affd700 1 -- 172.21.15.113:0/1623512770 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa45c0cc9c0 con 0x7fa43c06b350 2024-01-16T13:53:13.958 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-16T13:53:13.958 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:13.958 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "cephfs" 2024-01-16T13:53:13.958 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:13.958 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa43c06b350 msgr2=0x7fa43c06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa43c06b350 0x7fa43c06d810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa45c1f6d10 tx=0x7fa44c009040 comp rx=0 tx=0).stop 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 msgr2=0x7fa45c0cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0cb940 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa45000e760 tx=0x7fa45000eb20 comp rx=0 tx=0).stop 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 shutdown_connections 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fa43c06b350 0x7fa43c06d810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa45c15e1e0 0x7fa45c0ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa45c0d0f80 0x7fa45c0cb940 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 --2- 172.21.15.113:0/1623512770 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa45c0d3a00 0x7fa45c1f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:13.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 >> 172.21.15.113:0/1623512770 conn(0x7fa45c0b5b50 msgr2=0x7fa45c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:13.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 shutdown_connections 2024-01-16T13:53:13.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:13.978+0000 7fa460fa1700 1 -- 172.21.15.113:0/1623512770 wait complete. 2024-01-16T13:53:13.972 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T13:53:14.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.114+0000 7f25fb4e1700 1 Processor -- start 2024-01-16T13:53:14.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.114+0000 7f25fb4e1700 1 -- start start 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25fb4e1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 0x7f25f40cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25fb4e1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f25f40b98e0 0x7f25f40cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25fb4e1700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f4162af0 con 0x7f25f40b59d0 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25fb4e1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f40d13e0 con 0x7f25f40b98e0 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25fb4e1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4161f70 con 0x7f25f4161510 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25f9a7e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f25f40b59d0 0x7f25f40b5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:49730/0 (socket says 172.21.15.113:49730) 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25f9a7e700 1 -- 172.21.15.113:0/986111492 learned_addr learned my addr 172.21.15.113:0/986111492 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25f927d700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 0x7f25f40cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25f8a7c700 1 --2- 172.21.15.113:0/986111492 >> v2:172.21.15.124:3300/0 conn(0x7f25f40b98e0 0x7f25f40cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691900169 0 0) 0x7f25f4162af0 con 0x7f25f40b59d0 2024-01-16T13:53:14.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e00035b0 con 0x7f25f40b59d0 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975440673 0 0) 0x7f25e00035b0 con 0x7f25f40b59d0 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 msgr2=0x7f25f40cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 0x7f25f40cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 >> v2:172.21.15.124:3300/0 conn(0x7f25f40b98e0 msgr2=0x7f25f40cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 --2- 172.21.15.113:0/986111492 >> v2:172.21.15.124:3300/0 conn(0x7f25f40b98e0 0x7f25f40cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f4162250 con 0x7f25f40b59d0 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f25ec0019f0 con 0x7f25f40b59d0 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 >> v1:172.21.15.113:6789/0 conn(0x7f25f40b59d0 legacy=0x7f25f40b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 0x7f25e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.115+0000 7f25ebfff700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25e0004d90 0x7f25e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f40d13e0 con 0x7f25e0003a00 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4161f70 con 0x7f25e0004d90 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e000ba10 con 0x7f25f4161510 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f8a7c700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 0x7f25e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f9a7e700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25e0004d90 0x7f25e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 -- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 msgr2=0x7f25e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f4161510 0x7f25e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 -- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25e0004d90 msgr2=0x7f25e0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25e0004d90 0x7f25e0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 -- 172.21.15.113:0/986111492 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e00035b0 con 0x7f25e0003a00 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25f927d700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25e0003e20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009b60 tx=0x7f25e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4e9e5eddc7c892 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f000e6c0 con 0x7f25e0003a00 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f000e820 con 0x7f25e0003a00 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.116+0000 7f25ebfff700 1 -- 172.21.15.113:0/986111492 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f000eb90 con 0x7f25e0003a00 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25fb4e1700 1 -- 172.21.15.113:0/986111492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 msgr2=0x7f25e0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/986111492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25e0003e20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009b60 tx=0x7f25e0005880 comp rx=0 tx=0).stop 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25f9a7e700 1 -- 172.21.15.113:0/986111492 reap_dead start 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25fb4e1700 1 -- 172.21.15.113:0/986111492 shutdown_connections 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25fb4e1700 1 -- 172.21.15.113:0/986111492 >> 172.21.15.113:0/986111492 conn(0x7f25f406b410 msgr2=0x7f25f406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:14.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25fb4e1700 1 -- 172.21.15.113:0/986111492 shutdown_connections 2024-01-16T13:53:14.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.117+0000 7f25fb4e1700 1 -- 172.21.15.113:0/986111492 wait complete. 2024-01-16T13:53:14.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.118+0000 7f25fb4e1700 1 Processor -- start 2024-01-16T13:53:14.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.118+0000 7f25fb4e1700 1 -- start start 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25f40c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25f40cb020 0x7f25f40cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f415d0c0 0x7f25f415d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f415e990 con 0x7f25e0003a00 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f40cb9e0 con 0x7f25f40cb020 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f40ca280 con 0x7f25f415d0c0 2024-01-16T13:53:14.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25f40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25f40c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59866/0 (socket says 172.21.15.113:59866) 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 -- 172.21.15.113:0/2885461414 learned_addr learned my addr 172.21.15.113:0/2885461414 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f8a7c700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25f40cb020 0x7f25f40cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f9a7e700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f415d0c0 0x7f25f415d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 -- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f415d0c0 msgr2=0x7f25f415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f415d0c0 0x7f25f415d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 -- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25f40cb020 msgr2=0x7f25f40cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25f40cb020 0x7f25f40cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 -- 172.21.15.113:0/2885461414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f415da80 con 0x7f25e0003a00 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25f927d700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25f40c9d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f25e0004730 tx=0x7f25f00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f000e6c0 con 0x7f25e0003a00 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f0003ab0 con 0x7f25e0003a00 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f415dd10 con 0x7f25e0003a00 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.119+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0003e60 con 0x7f25e0003a00 2024-01-16T13:53:14.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.120+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f415e2d0 con 0x7f25e0003a00 2024-01-16T13:53:14.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.120+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f4068450 con 0x7f25e0003a00 2024-01-16T13:53:14.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.121+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25f0013470 con 0x7f25e0003a00 2024-01-16T13:53:14.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.121+0000 7f25e9ffb700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f25d806b350 0x7f25d806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.121+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25f008b640 con 0x7f25e0003a00 2024-01-16T13:53:14.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.121+0000 7f25f8a7c700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f25d806b350 0x7f25d806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.122+0000 7f25f8a7c700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f25d806b350 0x7f25d806d810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f25f40d29a0 tx=0x7f25e4009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.126+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25f00565c0 con 0x7f25e0003a00 2024-01-16T13:53:14.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.316+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f25f4069020 con 0x7f25e0003a00 2024-01-16T13:53:14.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.316+0000 7f25e9ffb700 1 -- 172.21.15.113:0/2885461414 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f25f0059be0 con 0x7f25e0003a00 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f25d806b350 msgr2=0x7f25d806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f25d806b350 0x7f25d806d810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f25f40d29a0 tx=0x7f25e4009210 comp rx=0 tx=0).stop 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 msgr2=0x7f25f40c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25f40c9d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f25e0004730 tx=0x7f25f00047c0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 shutdown_connections 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f25d806b350 0x7f25d806d810 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25f415d0c0 0x7f25f415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25e0003a00 0x7f25f40c9d40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 --2- 172.21.15.113:0/2885461414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25f40cb020 0x7f25f40cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 >> 172.21.15.113:0/2885461414 conn(0x7f25f406b410 msgr2=0x7f25f40b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 shutdown_connections 2024-01-16T13:53:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.318+0000 7f25fb4e1700 1 -- 172.21.15.113:0/2885461414 wait complete. 2024-01-16T13:53:14.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T13:53:14.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 Processor -- start 2024-01-16T13:53:14.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 -- start start 2024-01-16T13:53:14.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fda780b4ed0 0x7fda7806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 0x7fda7815e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7806f330 con 0x7fda7815e9b0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7806c9f0 con 0x7fda780b4ed0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.456+0000 7fda7d06d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7815f250 con 0x7fda7806c540 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda7759e700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fda7815e9b0 0x7fda78073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:49736/0 (socket says 172.21.15.113:49736) 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda7759e700 1 -- 172.21.15.113:0/1999997811 learned_addr learned my addr 172.21.15.113:0/1999997811 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda76d9d700 1 --2- 172.21.15.113:0/1999997811 >> v2:172.21.15.124:3300/0 conn(0x7fda780b4ed0 0x7fda7806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda7659c700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 0x7fda7815e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682120196 0 0) 0x7fda7806f330 con 0x7fda7815e9b0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda600035b0 con 0x7fda7815e9b0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 579334132 0 0) 0x7fda600035b0 con 0x7fda7815e9b0 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 msgr2=0x7fda7815e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 0x7fda7815e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 >> v2:172.21.15.124:3300/0 conn(0x7fda780b4ed0 msgr2=0x7fda7806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 --2- 172.21.15.113:0/1999997811 >> v2:172.21.15.124:3300/0 conn(0x7fda780b4ed0 0x7fda7806c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda78161a60 con 0x7fda7815e9b0 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fda700019f0 con 0x7fda7815e9b0 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 >> v1:172.21.15.113:6789/0 conn(0x7fda7815e9b0 legacy=0x7fda78073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda60003a00 0x7fda60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda60004850 0x7fda60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 0x7fda60005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7806c9f0 con 0x7fda60004850 2024-01-16T13:53:14.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7815f250 con 0x7fda60003a00 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6000ba60 con 0x7fda7806c540 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda7659c700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda60004850 0x7fda60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda76d9d700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda60003a00 0x7fda60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.457+0000 7fda7759e700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 0x7fda60005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7659c700 1 -- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 msgr2=0x7fda60005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7659c700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda7806c540 0x7fda60005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7659c700 1 -- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda60003a00 msgr2=0x7fda60003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7659c700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda60003a00 0x7fda60003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7659c700 1 -- 172.21.15.113:0/1999997811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda600035b0 con 0x7fda60004850 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7659c700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda60004850 0x7fda60004cd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009b60 tx=0x7fda60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41958129f650dc5d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c00e6c0 con 0x7fda60004850 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c00e820 con 0x7fda60004850 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda75d9b700 1 -- 172.21.15.113:0/1999997811 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c00eb90 con 0x7fda60004850 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7d06d700 1 -- 172.21.15.113:0/1999997811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda60004850 msgr2=0x7fda60004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7d06d700 1 --2- 172.21.15.113:0/1999997811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda60004850 0x7fda60004cd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009b60 tx=0x7fda60005880 comp rx=0 tx=0).stop 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7759e700 1 -- 172.21.15.113:0/1999997811 reap_dead start 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7d06d700 1 -- 172.21.15.113:0/1999997811 shutdown_connections 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7d06d700 1 -- 172.21.15.113:0/1999997811 >> 172.21.15.113:0/1999997811 conn(0x7fda78071b70 msgr2=0x7fda78071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7d06d700 1 -- 172.21.15.113:0/1999997811 shutdown_connections 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.458+0000 7fda7d06d700 1 -- 172.21.15.113:0/1999997811 wait complete. 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.460+0000 7fda7d06d700 1 Processor -- start 2024-01-16T13:53:14.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.460+0000 7fda7d06d700 1 -- start start 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda60004850 0x7fda7814a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda7815dbc0 0x7fda78159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 0x7fda78156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda78157170 con 0x7fda781567b0 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda78159dd0 con 0x7fda7815dbc0 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7815a0b0 con 0x7fda60004850 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 0x7fda78156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 0x7fda78156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59878/0 (socket says 172.21.15.113:59878) 2024-01-16T13:53:14.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 -- 172.21.15.113:0/173471407 learned_addr learned my addr 172.21.15.113:0/173471407 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda76d9d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda60004850 0x7fda7814a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7659c700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda7815dbc0 0x7fda78159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 -- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda60004850 msgr2=0x7fda7814a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda60004850 0x7fda7814a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 -- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda7815dbc0 msgr2=0x7fda78159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda7815dbc0 0x7fda78159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 -- 172.21.15.113:0/173471407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7815a390 con 0x7fda781567b0 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7759e700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 0x7fda78156c30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda78158ac0 tx=0x7fda7000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000ad00 con 0x7fda781567b0 2024-01-16T13:53:14.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000ae60 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7815a670 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda700135f0 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.461+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda78159a50 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.462+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda78162820 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.463+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fda70013750 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.463+0000 7fda5f7fe700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda6406b350 0x7fda6406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.463+0000 7fda76d9d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda6406b350 0x7fda6406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.463+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda7008a620 con 0x7fda781567b0 2024-01-16T13:53:14.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.464+0000 7fda76d9d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda6406b350 0x7fda6406d810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fda68009c60 tx=0x7fda68009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.468+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda70055520 con 0x7fda781567b0 2024-01-16T13:53:14.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.692+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fda7815a9e0 con 0x7fda6406b350 2024-01-16T13:53:14.678 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:14.696+0000 7fa588d60700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:14.678 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:14.696+0000 7fa588d60700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:14.678 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:14.696+0000 7fa588d60700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:14.678 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:14.696+0000 7fa588d60700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:14.678 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:14.696+0000 7fa588d60700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:14.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.730+0000 7fda5f7fe700 1 -- 172.21.15.113:0/173471407 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fda7815a9e0 con 0x7fda6406b350 2024-01-16T13:53:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda6406b350 msgr2=0x7fda6406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda6406b350 0x7fda6406d810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fda68009c60 tx=0x7fda68009210 comp rx=0 tx=0).stop 2024-01-16T13:53:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 msgr2=0x7fda78156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 0x7fda78156c30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda78158ac0 tx=0x7fda7000a5f0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 shutdown_connections 2024-01-16T13:53:14.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fda6406b350 0x7fda6406d810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda781567b0 0x7fda78156c30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fda60004850 0x7fda7814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 --2- 172.21.15.113:0/173471407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fda7815dbc0 0x7fda78159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 >> 172.21.15.113:0/173471407 conn(0x7fda78071b70 msgr2=0x7fda780b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 shutdown_connections 2024-01-16T13:53:14.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.732+0000 7fda7d06d700 1 -- 172.21.15.113:0/173471407 wait complete. 2024-01-16T13:53:14.726 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T13:53:14.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.868+0000 7fbc34353700 1 Processor -- start 2024-01-16T13:53:14.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.868+0000 7fbc34353700 1 -- start start 2024-01-16T13:53:14.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc34353700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc2c0b5220 0x7fbc2c0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc34353700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbc2c073f90 0x7fbc2c06e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc34353700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c0b42a0 con 0x7fbc2c15e280 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc34353700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0af900 con 0x7fbc2c0b5220 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc34353700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c074d30 con 0x7fbc2c073f90 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc2c0b5220 0x7fbc2c0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbc2c0b5220 0x7fbc2c0b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59904/0 (socket says 172.21.15.113:59904) 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 -- 172.21.15.113:0/398299642 learned_addr learned my addr 172.21.15.113:0/398299642 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc328f0700 1 --2- 172.21.15.113:0/398299642 >> v2:172.21.15.124:3300/0 conn(0x7fbc2c073f90 0x7fbc2c06e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 -- 172.21.15.113:0/398299642 >> v2:172.21.15.124:3300/0 conn(0x7fbc2c073f90 msgr2=0x7fbc2c06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 --2- 172.21.15.113:0/398299642 >> v2:172.21.15.124:3300/0 conn(0x7fbc2c073f90 0x7fbc2c06e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 -- 172.21.15.113:0/398299642 >> v1:172.21.15.113:6789/0 conn(0x7fbc2c15e280 legacy=0x7fbc2c0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 -- 172.21.15.113:0/398299642 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c074490 con 0x7fbc2c0b5220 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.869+0000 7fbc318ee700 1 --2- 172.21.15.113:0/398299642 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0b3d60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c071bc0 tx=0x7fbc28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851e625a071ed01b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc310ed700 1 -- 172.21.15.113:0/398299642 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2800b870 con 0x7fbc2c0b5220 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc310ed700 1 -- 172.21.15.113:0/398299642 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2800b9d0 con 0x7fbc2c0b5220 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc310ed700 1 -- 172.21.15.113:0/398299642 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2800bd60 con 0x7fbc2c0b5220 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 -- 172.21.15.113:0/398299642 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 msgr2=0x7fbc2c0b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 --2- 172.21.15.113:0/398299642 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0b3d60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c071bc0 tx=0x7fbc28009bd0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 -- 172.21.15.113:0/398299642 shutdown_connections 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 --2- 172.21.15.113:0/398299642 >> v2:172.21.15.124:3300/0 conn(0x7fbc2c073f90 0x7fbc2c06e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 --2- 172.21.15.113:0/398299642 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0b3d60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 -- 172.21.15.113:0/398299642 >> 172.21.15.113:0/398299642 conn(0x7fbc2c072b20 msgr2=0x7fbc2c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 -- 172.21.15.113:0/398299642 shutdown_connections 2024-01-16T13:53:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.870+0000 7fbc34353700 1 -- 172.21.15.113:0/398299642 wait complete. 2024-01-16T13:53:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.871+0000 7fbc34353700 1 Processor -- start 2024-01-16T13:53:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 -- start start 2024-01-16T13:53:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc2c073f90 0x7fbc2c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc2c147e30 0x7fbc2c1482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15acc0 con 0x7fbc2c0b5220 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c1487f0 con 0x7fbc2c147e30 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc34353700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15a040 con 0x7fbc2c073f90 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc318ee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc318ee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0c5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59908/0 (socket says 172.21.15.113:59908) 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.872+0000 7fbc318ee700 1 -- 172.21.15.113:0/3892025813 learned_addr learned my addr 172.21.15.113:0/3892025813 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc328f0700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc2c147e30 0x7fbc2c1482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc318ee700 1 -- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc2c073f90 msgr2=0x7fbc2c160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc318ee700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc2c073f90 0x7fbc2c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc318ee700 1 -- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc2c147e30 msgr2=0x7fbc2c1482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc318ee700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc2c147e30 0x7fbc2c1482b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc318ee700 1 -- 172.21.15.113:0/3892025813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc28009550 con 0x7fbc2c0b5220 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc328f0700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc2c147e30 0x7fbc2c1482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc318ee700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0c5cd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbc28000c00 tx=0x7fbc28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28004540 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc280036a0 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c15a320 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28003a30 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.873+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c15a8e0 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.874+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c15aa70 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.874+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbc28021070 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.875+0000 7fbc22ffd700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1006b350 0x7fbc1006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.875+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc2808b3b0 con 0x7fbc2c0b5220 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.875+0000 7fbc320ef700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1006b350 0x7fbc1006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:14.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.875+0000 7fbc320ef700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1006b350 0x7fbc1006d810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c000f60 tx=0x7fbc1c007a40 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:14.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:14.879+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc280563d0 con 0x7fbc2c0b5220 2024-01-16T13:53:15.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.103+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc2c06dd00 con 0x7fbc1006b350 2024-01-16T13:53:15.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.108+0000 7fbc22ffd700 1 -- 172.21.15.113:0/3892025813 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fbc1c005080 con 0x7fbc1006b350 2024-01-16T13:53:15.090 INFO:teuthology.orchestra.run.smithi113.stdout:/volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7 2024-01-16T13:53:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1006b350 msgr2=0x7fbc1006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1006b350 0x7fbc1006d810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c000f60 tx=0x7fbc1c007a40 comp rx=0 tx=0).stop 2024-01-16T13:53:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 msgr2=0x7fbc2c0c5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0c5cd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbc28000c00 tx=0x7fbc28004750 comp rx=0 tx=0).stop 2024-01-16T13:53:15.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 shutdown_connections 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbc1006b350 0x7fbc1006d810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc2c147e30 0x7fbc2c1482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc2c073f90 0x7fbc2c160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 --2- 172.21.15.113:0/3892025813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbc2c0b5220 0x7fbc2c0c5cd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 >> 172.21.15.113:0/3892025813 conn(0x7fbc2c072b20 msgr2=0x7fbc2c0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 shutdown_connections 2024-01-16T13:53:15.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:15.110+0000 7fbc34353700 1 -- 172.21.15.113:0/3892025813 wait complete. 2024-01-16T13:53:15.104 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7 2024-01-16T13:53:15.104 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:53:15.104 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:15.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:15.121 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T13:53:20.638 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records in 2024-01-16T13:53:20.638 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records out 2024-01-16T13:53:20.638 INFO:teuthology.orchestra.run.smithi113.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.45088 s, 197 MB/s 2024-01-16T13:53:20.638 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:53:20.638 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:20.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:20.657 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T13:53:26.182 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records in 2024-01-16T13:53:26.183 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records out 2024-01-16T13:53:26.183 INFO:teuthology.orchestra.run.smithi113.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.46828 s, 196 MB/s 2024-01-16T13:53:26.184 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:53:26.184 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:26.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:26.201 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T13:53:31.776 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records in 2024-01-16T13:53:31.776 INFO:teuthology.orchestra.run.smithi113.stderr:1024+0 records out 2024-01-16T13:53:31.776 INFO:teuthology.orchestra.run.smithi113.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.51646 s, 195 MB/s 2024-01-16T13:53:31.777 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T13:53:31.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.919+0000 7f1c57027700 1 Processor -- start 2024-01-16T13:53:31.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.919+0000 7f1c57027700 1 -- start start 2024-01-16T13:53:31.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c57027700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c500b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c57027700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1c500d0630 0x7f1c5006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c57027700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c500ae300 con 0x7f1c50073f90 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c57027700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5015fb00 con 0x7f1c500d0630 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c57027700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5015c410 con 0x7f1c5015c030 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c4ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1c500d0630 0x7f1c5006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c4ffff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1c500d0630 0x7f1c5006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59394/0 (socket says 172.21.15.113:59394) 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.920+0000 7f1c4ffff700 1 -- 172.21.15.113:0/1764262745 learned_addr learned my addr 172.21.15.113:0/1764262745 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c54dc3700 1 --2- 172.21.15.113:0/1764262745 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c500b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4ffff700 1 -- 172.21.15.113:0/1764262745 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 msgr2=0x7f1c500b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4ffff700 1 --2- 172.21.15.113:0/1764262745 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c500b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4ffff700 1 -- 172.21.15.113:0/1764262745 >> v1:172.21.15.113:6789/0 conn(0x7f1c50073f90 legacy=0x7f1c5015ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4ffff700 1 -- 172.21.15.113:0/1764262745 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5006c550 con 0x7f1c500d0630 2024-01-16T13:53:31.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4ffff700 1 --2- 172.21.15.113:0/1764262745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c500d0630 0x7f1c5006dc40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0098f0 tx=0x7f1c3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1f28dbe6eb269d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4f7fe700 1 -- 172.21.15.113:0/1764262745 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c011830 con 0x7f1c500d0630 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4f7fe700 1 -- 172.21.15.113:0/1764262745 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c011990 con 0x7f1c500d0630 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.921+0000 7f1c4f7fe700 1 -- 172.21.15.113:0/1764262745 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c011d20 con 0x7f1c500d0630 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 -- 172.21.15.113:0/1764262745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c500d0630 msgr2=0x7f1c5006dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 --2- 172.21.15.113:0/1764262745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c500d0630 0x7f1c5006dc40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0098f0 tx=0x7f1c3c009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 -- 172.21.15.113:0/1764262745 shutdown_connections 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 --2- 172.21.15.113:0/1764262745 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c500b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 --2- 172.21.15.113:0/1764262745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c500d0630 0x7f1c5006dc40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 -- 172.21.15.113:0/1764262745 >> 172.21.15.113:0/1764262745 conn(0x7f1c500d0c00 msgr2=0x7f1c500d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 -- 172.21.15.113:0/1764262745 shutdown_connections 2024-01-16T13:53:31.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.922+0000 7f1c57027700 1 -- 172.21.15.113:0/1764262745 wait complete. 2024-01-16T13:53:31.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.923+0000 7f1c57027700 1 Processor -- start 2024-01-16T13:53:31.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.923+0000 7f1c57027700 1 -- start start 2024-01-16T13:53:31.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 0x7f1c5014cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c500d0630 0x7f1c501f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c50152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50150410 con 0x7f1c50073f90 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5014d6d0 con 0x7f1c500d0630 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5014fa70 con 0x7f1c5015c030 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 0x7f1c5014cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 0x7f1c5014cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59396/0 (socket says 172.21.15.113:59396) 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 -- 172.21.15.113:0/1156074021 learned_addr learned my addr 172.21.15.113:0/1156074021 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c4ffff700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c500d0630 0x7f1c501f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c555c4700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c50152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 -- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 msgr2=0x7f1c50152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c50152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 -- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c500d0630 msgr2=0x7f1c501f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c500d0630 0x7f1c501f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:31.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 -- 172.21.15.113:0/1156074021 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c009550 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c555c4700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c50152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c54dc3700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 0x7f1c5014cdd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400e870 tx=0x7f1c4400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c44009590 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c44013680 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5014fd50 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c440139f0 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.924+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5014d310 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.925+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c50150550 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.926+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f1c44013b50 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.926+0000 7f1c4d7fa700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1c3406b3e0 0x7f1c3406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.926+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c44089a20 con 0x7f1c50073f90 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.926+0000 7f1c4ffff700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1c3406b3e0 0x7f1c3406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:31.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.927+0000 7f1c4ffff700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1c3406b3e0 0x7f1c3406d8a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009c40 tx=0x7f1c3c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:31.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:31.931+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c50150550 con 0x7f1c50073f90 2024-01-16T13:53:32.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.121+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f1c50150550 con 0x7f1c50073f90 2024-01-16T13:53:32.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.124+0000 7f1c4d7fa700 1 -- 172.21.15.113:0/1156074021 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f1c50150550 con 0x7f1c50073f90 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1c3406b3e0 msgr2=0x7f1c3406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1c3406b3e0 0x7f1c3406d8a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009c40 tx=0x7f1c3c008030 comp rx=0 tx=0).stop 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 msgr2=0x7f1c5014cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 0x7f1c5014cdd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400e870 tx=0x7f1c4400eb80 comp rx=0 tx=0).stop 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 shutdown_connections 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f1c3406b3e0 0x7f1c3406d8a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c5015c030 0x7f1c50152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c50073f90 0x7f1c5014cdd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 --2- 172.21.15.113:0/1156074021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c500d0630 0x7f1c501f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 >> 172.21.15.113:0/1156074021 conn(0x7f1c500d0c00 msgr2=0x7f1c5006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 shutdown_connections 2024-01-16T13:53:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.127+0000 7f1c57027700 1 -- 172.21.15.113:0/1156074021 wait complete. 2024-01-16T13:53:32.122 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T13:53:32.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.266+0000 7f788cbd0700 1 Processor -- start 2024-01-16T13:53:32.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.266+0000 7f788cbd0700 1 -- start start 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f788cbd0700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7888073f90 0x7f7888074440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f788cbd0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f788cbd0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78880b0590 con 0x7f788815e2a0 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f788cbd0700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7888074980 con 0x7f7888073f90 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f788cbd0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78880d0690 con 0x7f7888071c00 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7885d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7888073f90 0x7f7888074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7885d9b700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7888073f90 0x7f7888074440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59408/0 (socket says 172.21.15.113:59408) 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7885d9b700 1 -- 172.21.15.113:0/4225596976 learned_addr learned my addr 172.21.15.113:0/4225596976 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7886d9d700 1 --2- 172.21.15.113:0/4225596976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7885d9b700 1 -- 172.21.15.113:0/4225596976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 msgr2=0x7f7888072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7885d9b700 1 --2- 172.21.15.113:0/4225596976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f7885d9b700 1 -- 172.21.15.113:0/4225596976 >> v1:172.21.15.113:6789/0 conn(0x7f788815e2a0 legacy=0x7f78880afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f7885d9b700 1 -- 172.21.15.113:0/4225596976 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78880cf7e0 con 0x7f7888073f90 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.267+0000 7f788559a700 1 -- 172.21.15.113:0/4225596976 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318594151 0 0) 0x7f78880b0590 con 0x7f788815e2a0 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f7885d9b700 1 --2- 172.21.15.113:0/4225596976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888073f90 0x7f7888074440 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f78780098f0 tx=0x7f7878009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ec9f6677fb430f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788559a700 1 -- 172.21.15.113:0/4225596976 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787800b8a0 con 0x7f7888073f90 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788559a700 1 -- 172.21.15.113:0/4225596976 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787800ba00 con 0x7f7888073f90 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788559a700 1 -- 172.21.15.113:0/4225596976 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787800bd90 con 0x7f7888073f90 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 -- 172.21.15.113:0/4225596976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888073f90 msgr2=0x7f7888074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 --2- 172.21.15.113:0/4225596976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888073f90 0x7f7888074440 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f78780098f0 tx=0x7f7878009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 -- 172.21.15.113:0/4225596976 shutdown_connections 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 --2- 172.21.15.113:0/4225596976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 --2- 172.21.15.113:0/4225596976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888073f90 0x7f7888074440 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 -- 172.21.15.113:0/4225596976 >> 172.21.15.113:0/4225596976 conn(0x7f78880d0af0 msgr2=0x7f78880d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 -- 172.21.15.113:0/4225596976 shutdown_connections 2024-01-16T13:53:32.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.268+0000 7f788cbd0700 1 -- 172.21.15.113:0/4225596976 wait complete. 2024-01-16T13:53:32.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.270+0000 7f788cbd0700 1 Processor -- start 2024-01-16T13:53:32.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.270+0000 7f788cbd0700 1 -- start start 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7888073f90 0x7f78880b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 0x7f7888160760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788815e680 con 0x7f788815e2a0 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78881f3ae0 con 0x7f7888073f90 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78880b0ed0 con 0x7f7888071c00 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 0x7f7888160760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 0x7f7888160760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59418/0 (socket says 172.21.15.113:59418) 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 -- 172.21.15.113:0/1013602639 learned_addr learned my addr 172.21.15.113:0/1013602639 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7885d9b700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7888073f90 0x7f78880b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788659c700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 -- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 msgr2=0x7f7888155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 -- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7888073f90 msgr2=0x7f78880b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7888073f90 0x7f78880b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 -- 172.21.15.113:0/1013602639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878009550 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7886d9d700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 0x7f7888160760 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7880007e40 tx=0x7f788000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788000ed30 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880014420 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78880b11b0 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78800147d0 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.271+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788815e920 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.273+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888067f60 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.273+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f7880014930 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.273+0000 7f7876ffd700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f786806b3e0 0x7f786806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.273+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f788008a1d0 con 0x7f788815e2a0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.273+0000 7f788659c700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f786806b3e0 0x7f786806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.274+0000 7f788659c700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f786806b3e0 0x7f786806d8a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f787c000f60 tx=0x7f787c009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:32.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.278+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78800550e0 con 0x7f788815e2a0 2024-01-16T13:53:32.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.506+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f78880d0040 con 0x7f786806b3e0 2024-01-16T13:53:32.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.521+0000 7f7876ffd700 1 -- 172.21.15.113:0/1013602639 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f787c0014e0 con 0x7f786806b3e0 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f786806b3e0 msgr2=0x7f786806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f786806b3e0 0x7f786806d8a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f787c000f60 tx=0x7f787c009040 comp rx=0 tx=0).stop 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 msgr2=0x7f7888160760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 0x7f7888160760 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7880007e40 tx=0x7f788000e3c0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 shutdown_connections 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f786806b3e0 0x7f786806d8a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f788815e2a0 0x7f7888160760 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7888071c00 0x7f7888155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 --2- 172.21.15.113:0/1013602639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7888073f90 0x7f78880b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 >> 172.21.15.113:0/1013602639 conn(0x7f78880d0af0 msgr2=0x7f78880ad900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:32.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 shutdown_connections 2024-01-16T13:53:32.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.523+0000 7f788cbd0700 1 -- 172.21.15.113:0/1013602639 wait complete. 2024-01-16T13:53:32.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T13:53:32.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.660+0000 7f9e0f0f0700 1 Processor -- start 2024-01-16T13:53:32.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.660+0000 7f9e0f0f0700 1 -- start start 2024-01-16T13:53:32.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e0f0f0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9e0806c710 0x7f9e0806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e0f0f0700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080b4940 con 0x7f9e080ae750 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e0f0f0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080b4c20 con 0x7f9e08147840 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e0f0f0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e080b4f00 con 0x7f9e0806c710 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e0d68d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f9e080ae750 0x7f9e0806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:53932/0 (socket says 172.21.15.113:53932) 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e0d68d700 1 -- 172.21.15.113:0/1997294693 learned_addr learned my addr 172.21.15.113:0/1997294693 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e07fff700 1 --2- 172.21.15.113:0/1997294693 >> v2:172.21.15.124:3300/0 conn(0x7f9e0806c710 0x7f9e0806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639685407 0 0) 0x7f9e080b4940 con 0x7f9e080ae750 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df80035b0 con 0x7f9e080ae750 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008777025 0 0) 0x7f9df80035b0 con 0x7f9e080ae750 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 >> v2:172.21.15.124:3300/0 conn(0x7f9e0806c710 msgr2=0x7f9e0806d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 --2- 172.21.15.113:0/1997294693 >> v2:172.21.15.124:3300/0 conn(0x7f9e0806c710 0x7f9e0806d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 >> v1:172.21.15.124:6789/0 conn(0x7f9e08147840 legacy=0x7f9e0806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080b51e0 con 0x7f9e080ae750 2024-01-16T13:53:32.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7f9e000019f0 con 0x7f9e080ae750 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 >> v1:172.21.15.113:6789/0 conn(0x7f9e080ae750 legacy=0x7f9e0806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.661+0000 7f9e077fe700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9df8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9df8004850 0x7f9df8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9df8005210 0x7f9df800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080b4f00 con 0x7f9df8003a00 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df800c0b0 con 0x7f9df8005210 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df800c270 con 0x7f9df8004850 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9df8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0d68d700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9df8005210 0x7f9df800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e07fff700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9df8004850 0x7f9df8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9df8004850 msgr2=0x7f9df8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9df8004850 0x7f9df8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9df8005210 msgr2=0x7f9df800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9df8005210 0x7f9df800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/1997294693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80035b0 con 0x7f9df8003a00 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0d68d700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9df8005210 0x7f9df800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0d68d700 1 -- 172.21.15.113:0/1997294693 reap_dead start 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9df8003e20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0017c0 tx=0x7f9df8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28cb214f3cc3bafa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0052d0 con 0x7f9df8003a00 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc005430 con 0x7f9df8003a00 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.662+0000 7f9e077fe700 1 -- 172.21.15.113:0/1997294693 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0038c0 con 0x7f9df8003a00 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/1997294693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 msgr2=0x7f9df8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9df8003e20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0017c0 tx=0x7f9df8005880 comp rx=0 tx=0).stop 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/1997294693 shutdown_connections 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/1997294693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9df8003e20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/1997294693 >> 172.21.15.113:0/1997294693 conn(0x7f9e08071c70 msgr2=0x7f9e08072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/1997294693 shutdown_connections 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.663+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/1997294693 wait complete. 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.664+0000 7f9e0f0f0700 1 Processor -- start 2024-01-16T13:53:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.664+0000 7f9e0f0f0700 1 -- start start 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0f0f0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9e08157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0f0f0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9e08142720 0x7f9e080c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0f0f0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9e080c9170 0x7f9e080c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0f0f0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080c6d10 con 0x7f9df8003a00 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0f0f0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080c9b40 con 0x7f9e080c9170 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0f0f0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08142cc0 con 0x7f9e08142720 2024-01-16T13:53:32.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9e08157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9e08157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59436/0 (socket says 172.21.15.113:59436) 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/3149589184 learned_addr learned my addr 172.21.15.113:0/3149589184 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0d68d700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9e080c9170 0x7f9e080c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e07fff700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9e08142720 0x7f9e080c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9e08142720 msgr2=0x7f9e080c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9e08142720 0x7f9e080c67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9e080c9170 msgr2=0x7f9e080c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9e080c9170 0x7f9e080c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 -- 172.21.15.113:0/3149589184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08142fa0 con 0x7f9df8003a00 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e0ce8c700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9e08157820 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005040 tx=0x7f9dfc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.665+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc001290 con 0x7f9df8003a00 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.666+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc01d480 con 0x7f9df8003a00 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.666+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08143280 con 0x7f9df8003a00 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.666+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc01d810 con 0x7f9df8003a00 2024-01-16T13:53:32.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.666+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e081f6f80 con 0x7f9df8003a00 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.667+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e081437a0 con 0x7f9df8003a00 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.667+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f9dfc019040 con 0x7f9df8003a00 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.667+0000 7f9e057fa700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9dec06b3e0 0x7f9dec06d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.668+0000 7f9e07fff700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9dec06b3e0 0x7f9dec06d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.668+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dfc08a740 con 0x7f9df8003a00 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.668+0000 7f9e07fff700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9dec06b3e0 0x7f9dec06d8a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009990 tx=0x7f9df4008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:32.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.672+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dfc055770 con 0x7f9df8003a00 2024-01-16T13:53:32.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.893+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e080c6d10 con 0x7f9dec06b3e0 2024-01-16T13:53:32.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.898+0000 7f9e057fa700 1 -- 172.21.15.113:0/3149589184 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9e080c6d10 con 0x7f9dec06b3e0 2024-01-16T13:53:32.881 INFO:teuthology.orchestra.run.smithi113.stdout:/volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7 2024-01-16T13:53:32.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9dec06b3e0 msgr2=0x7f9dec06d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9dec06b3e0 0x7f9dec06d8a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009990 tx=0x7f9df4008040 comp rx=0 tx=0).stop 2024-01-16T13:53:32.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 msgr2=0x7f9e08157820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:32.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9e08157820 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005040 tx=0x7f9dfc001120 comp rx=0 tx=0).stop 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 shutdown_connections 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9dec06b3e0 0x7f9dec06d8a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9e080c9170 0x7f9e080c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df8003a00 0x7f9e08157820 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 --2- 172.21.15.113:0/3149589184 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9e08142720 0x7f9e080c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 >> 172.21.15.113:0/3149589184 conn(0x7f9e08071c70 msgr2=0x7f9e0806d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 shutdown_connections 2024-01-16T13:53:32.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:32.900+0000 7f9e0f0f0700 1 -- 172.21.15.113:0/3149589184 wait complete. 2024-01-16T13:53:32.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:32.895 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7) 2024-01-16T13:53:32.913 INFO:teuthology.orchestra.run.smithi113.stdout:# file: volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7 2024-01-16T13:53:32.913 INFO:teuthology.orchestra.run.smithi113.stdout:ceph.dir.rbytes="2972172288" 2024-01-16T13:53:32.913 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:32.914 INFO:tasks.cephfs.test_volumes:r_size = 2972172288 2024-01-16T13:53:34.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:34.916 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7) 2024-01-16T13:53:34.938 INFO:teuthology.orchestra.run.smithi113.stdout:# file: volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7 2024-01-16T13:53:34.938 INFO:teuthology.orchestra.run.smithi113.stdout:ceph.dir.rbytes="2972172288" 2024-01-16T13:53:34.938 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:34.938 INFO:tasks.cephfs.test_volumes:r_size = 2972172288 2024-01-16T13:53:36.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:36.939 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7) 2024-01-16T13:53:36.961 INFO:teuthology.orchestra.run.smithi113.stdout:# file: volumes/_nogroup/sv1/f01c46ab-9f99-4eca-bfbd-c1b07f8f3cf7 2024-01-16T13:53:36.961 INFO:teuthology.orchestra.run.smithi113.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T13:53:36.961 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:36.962 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T13:53:36.962 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T13:53:37.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.158+0000 7f5d8db30700 1 Processor -- start 2024-01-16T13:53:37.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.158+0000 7f5d8db30700 1 -- start start 2024-01-16T13:53:37.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d8db30700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5d8806f250 0x7f5d8806f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d8db30700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5d8806fc50 0x7f5d88167b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d8db30700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8806b9a0 con 0x7f5d8815e1c0 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d8db30700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880d0ec0 con 0x7f5d8806fc50 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d8db30700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880700c0 con 0x7f5d8806f250 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5d8806fc50 0x7f5d88167b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5d8806fc50 0x7f5d88167b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37834/0 (socket says 172.21.15.113:37834) 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 -- 172.21.15.113:0/2012925706 learned_addr learned my addr 172.21.15.113:0/2012925706 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d86ffd700 1 --2- 172.21.15.113:0/2012925706 >> v2:172.21.15.124:3300/0 conn(0x7f5d8806f250 0x7f5d8806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 -- 172.21.15.113:0/2012925706 >> v2:172.21.15.124:3300/0 conn(0x7f5d8806f250 msgr2=0x7f5d8806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 --2- 172.21.15.113:0/2012925706 >> v2:172.21.15.124:3300/0 conn(0x7f5d8806f250 0x7f5d8806f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 -- 172.21.15.113:0/2012925706 >> v1:172.21.15.124:6789/0 conn(0x7f5d8815e1c0 legacy=0x7f5d880d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.159+0000 7f5d87fff700 1 -- 172.21.15.113:0/2012925706 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880d10a0 con 0x7f5d8806fc50 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d87fff700 1 --2- 172.21.15.113:0/2012925706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88167b70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5d880c5660 tx=0x7f5d8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57f5a3861fe6951 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d867fc700 1 -- 172.21.15.113:0/2012925706 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8000e070 con 0x7f5d8806fc50 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d867fc700 1 -- 172.21.15.113:0/2012925706 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80003a20 con 0x7f5d8806fc50 2024-01-16T13:53:37.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d867fc700 1 -- 172.21.15.113:0/2012925706 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003dd0 con 0x7f5d8806fc50 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 -- 172.21.15.113:0/2012925706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 msgr2=0x7f5d88167b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 --2- 172.21.15.113:0/2012925706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88167b70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5d880c5660 tx=0x7f5d8000b330 comp rx=0 tx=0).stop 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 -- 172.21.15.113:0/2012925706 shutdown_connections 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 --2- 172.21.15.113:0/2012925706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88167b70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 --2- 172.21.15.113:0/2012925706 >> v2:172.21.15.124:3300/0 conn(0x7f5d8806f250 0x7f5d8806f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 -- 172.21.15.113:0/2012925706 >> 172.21.15.113:0/2012925706 conn(0x7f5d880b3710 msgr2=0x7f5d880b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 -- 172.21.15.113:0/2012925706 shutdown_connections 2024-01-16T13:53:37.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.160+0000 7f5d8db30700 1 -- 172.21.15.113:0/2012925706 wait complete. 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.162+0000 7f5d8db30700 1 Processor -- start 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.162+0000 7f5d8db30700 1 -- start start 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.162+0000 7f5d8db30700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5d8806f250 0x7f5d880b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5d8815e1c0 0x7f5d88148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d881f3880 con 0x7f5d8806fc50 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880c9f30 con 0x7f5d8815e1c0 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880ca210 con 0x7f5d8806f250 2024-01-16T13:53:37.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37850/0 (socket says 172.21.15.113:37850) 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 -- 172.21.15.113:0/794523351 learned_addr learned my addr 172.21.15.113:0/794523351 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d87fff700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5d8815e1c0 0x7f5d88148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d877fe700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5d8806f250 0x7f5d880b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 -- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5d8806f250 msgr2=0x7f5d880b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5d8806f250 0x7f5d880b7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 -- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5d8815e1c0 msgr2=0x7f5d88148ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5d8815e1c0 0x7f5d88148ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 -- 172.21.15.113:0/794523351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80009b30 con 0x7f5d8806fc50 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d87fff700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5d8815e1c0 0x7f5d88148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d86ffd700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88148ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800c7f0 tx=0x7f5d7800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78011800 con 0x7f5d8806fc50 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78011960 con 0x7f5d8806fc50 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78011cf0 con 0x7f5d8806fc50 2024-01-16T13:53:37.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d880ca4f0 con 0x7f5d8806fc50 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.163+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d880caab0 con 0x7f5d8806fc50 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.164+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d88068480 con 0x7f5d8806fc50 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.165+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f5d78018360 con 0x7f5d8806fc50 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.165+0000 7f5d6ffff700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5d7406b3e0 0x7f5d7406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.165+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d78089ce0 con 0x7f5d8806fc50 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.165+0000 7f5d877fe700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5d7406b3e0 0x7f5d7406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.165+0000 7f5d877fe700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5d7406b3e0 0x7f5d7406d8a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c009870 tx=0x7f5d7c008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:37.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.170+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d780549d0 con 0x7f5d8806fc50 2024-01-16T13:53:37.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.400+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d8806b350 con 0x7f5d7406b3e0 2024-01-16T13:53:37.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.457+0000 7f5d6ffff700 1 -- 172.21.15.113:0/794523351 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5d7c0090c0 con 0x7f5d7406b3e0 2024-01-16T13:53:37.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5d7406b3e0 msgr2=0x7f5d7406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5d7406b3e0 0x7f5d7406d8a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c009870 tx=0x7f5d7c008040 comp rx=0 tx=0).stop 2024-01-16T13:53:37.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 msgr2=0x7f5d88148ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88148ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800c7f0 tx=0x7f5d7800cbb0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 shutdown_connections 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5d7406b3e0 0x7f5d7406d8a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5d8815e1c0 0x7f5d88148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5d8806f250 0x7f5d880b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 --2- 172.21.15.113:0/794523351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d8806fc50 0x7f5d88148ab0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 >> 172.21.15.113:0/794523351 conn(0x7f5d880b3710 msgr2=0x7f5d880b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 shutdown_connections 2024-01-16T13:53:37.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.459+0000 7f5d8db30700 1 -- 172.21.15.113:0/794523351 wait complete. 2024-01-16T13:53:37.444 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.460+0000 7fa58ed6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.444 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.460+0000 7fa58ed6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.444 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.460+0000 7fa58ed6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.444 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.460+0000 7fa58ed6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.444 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.460+0000 7fa58ed6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.456 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.473+0000 7fa58d569700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.456 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.473+0000 7fa58d569700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.456 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.473+0000 7fa58d569700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.457 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.473+0000 7fa58d569700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.457 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:37.473+0000 7fa58d569700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:37.457 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T13:53:37.457 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:37.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.603+0000 7fbe7e8e4700 1 Processor -- start 2024-01-16T13:53:37.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.603+0000 7fbe7e8e4700 1 -- start start 2024-01-16T13:53:37.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe7e8e4700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbe7815b240 0x7fbe7806e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe7e8e4700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbe7806c2e0 0x7fbe781647f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe7e8e4700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7806eeb0 con 0x7fbe7806e9a0 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe7e8e4700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7806f190 con 0x7fbe7815b240 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe7e8e4700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7806f470 con 0x7fbe7806c2e0 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe77fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbe7815b240 0x7fbe7806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe777fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fbe7806e9a0 0x7fbe7815b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:36408/0 (socket says 172.21.15.113:36408) 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.604+0000 7fbe777fe700 1 -- 172.21.15.113:0/1514387346 learned_addr learned my addr 172.21.15.113:0/1514387346 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe77fff700 1 -- 172.21.15.113:0/1514387346 >> v2:172.21.15.124:3300/0 conn(0x7fbe7806c2e0 msgr2=0x7fbe781647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe7ce81700 1 --2- 172.21.15.113:0/1514387346 >> v2:172.21.15.124:3300/0 conn(0x7fbe7806c2e0 0x7fbe781647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe77fff700 1 --2- 172.21.15.113:0/1514387346 >> v2:172.21.15.124:3300/0 conn(0x7fbe7806c2e0 0x7fbe781647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe77fff700 1 -- 172.21.15.113:0/1514387346 >> v1:172.21.15.113:6789/0 conn(0x7fbe7806e9a0 legacy=0x7fbe7815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe77fff700 1 -- 172.21.15.113:0/1514387346 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7806f750 con 0x7fbe7815b240 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe76ffd700 1 -- 172.21.15.113:0/1514387346 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746879616 0 0) 0x7fbe7806eeb0 con 0x7fbe7806e9a0 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe7ce81700 1 --2- 172.21.15.113:0/1514387346 >> v2:172.21.15.124:3300/0 conn(0x7fbe7806c2e0 0x7fbe781647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe77fff700 1 --2- 172.21.15.113:0/1514387346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7815b240 0x7fbe7806e460 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbe7809f070 tx=0x7fbe68009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25cbbce82ebb956 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe76ffd700 1 -- 172.21.15.113:0/1514387346 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68011830 con 0x7fbe7815b240 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe76ffd700 1 -- 172.21.15.113:0/1514387346 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe68011990 con 0x7fbe7815b240 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe76ffd700 1 -- 172.21.15.113:0/1514387346 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68011d20 con 0x7fbe7815b240 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/1514387346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7815b240 msgr2=0x7fbe7806e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.605+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/1514387346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7815b240 0x7fbe7806e460 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbe7809f070 tx=0x7fbe68009b20 comp rx=0 tx=0).stop 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.606+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/1514387346 shutdown_connections 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.606+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/1514387346 >> v2:172.21.15.124:3300/0 conn(0x7fbe7806c2e0 0x7fbe781647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.606+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/1514387346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7815b240 0x7fbe7806e460 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.606+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/1514387346 >> 172.21.15.113:0/1514387346 conn(0x7fbe780d0c00 msgr2=0x7fbe780d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:37.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.606+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/1514387346 shutdown_connections 2024-01-16T13:53:37.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.606+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/1514387346 wait complete. 2024-01-16T13:53:37.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.607+0000 7fbe7e8e4700 1 Processor -- start 2024-01-16T13:53:37.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.607+0000 7fbe7e8e4700 1 -- start start 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 0x7fbe78150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe7806e9a0 0x7fbe781f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe7815b240 0x7fbe7809cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780b09d0 con 0x7fbe7806c2e0 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780b0c90 con 0x7fbe7806e9a0 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780b0f70 con 0x7fbe7815b240 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 0x7fbe78150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 0x7fbe78150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37896/0 (socket says 172.21.15.113:37896) 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 -- 172.21.15.113:0/4276070779 learned_addr learned my addr 172.21.15.113:0/4276070779 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:37.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7ce81700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe7815b240 0x7fbe7809cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe777fe700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe7806e9a0 0x7fbe781f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 -- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe7815b240 msgr2=0x7fbe7809cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe7815b240 0x7fbe7809cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 -- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe7806e9a0 msgr2=0x7fbe781f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe7806e9a0 0x7fbe781f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 -- 172.21.15.113:0/4276070779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680094a0 con 0x7fbe7806c2e0 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7ce81700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe7815b240 0x7fbe7809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe777fe700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe7806e9a0 0x7fbe781f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe77fff700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 0x7fbe78150c40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbe680090b0 tx=0x7fbe68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe680038b0 con 0x7fbe7806c2e0 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6801f070 con 0x7fbe7806c2e0 2024-01-16T13:53:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe780b1250 con 0x7fbe7806c2e0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68003c70 con 0x7fbe7806c2e0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.608+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe78151180 con 0x7fbe7806c2e0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.609+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe780b1710 con 0x7fbe7806c2e0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.610+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fbe6801a430 con 0x7fbe7806c2e0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.610+0000 7fbe74ff9700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe5806b3e0 0x7fbe5806d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.610+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe68021070 con 0x7fbe7806c2e0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.610+0000 7fbe777fe700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe5806b3e0 0x7fbe5806d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:37.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.611+0000 7fbe777fe700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe5806b3e0 0x7fbe5806d8a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe7806df00 tx=0x7fbe60009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:37.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.615+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe68056760 con 0x7fbe7806c2e0 2024-01-16T13:53:37.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:37.846+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe7815bd30 con 0x7fbe5806b3e0 2024-01-16T13:53:39.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:39.561+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fbe68056b30 con 0x7fbe7806c2e0 2024-01-16T13:53:40.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:53:40.612 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:53:41.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.117+0000 7fbe74ff9700 1 -- 172.21.15.113:0/4276070779 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fbe60001cc0 con 0x7fbe5806b3e0 2024-01-16T13:53:41.100 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:41.100 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:41.100 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:41.100 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:41.100 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:41.101 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:41.101 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:41.101 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.101 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.101 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:41.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe5806b3e0 msgr2=0x7fbe5806d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe5806b3e0 0x7fbe5806d8a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe7806df00 tx=0x7fbe60009040 comp rx=0 tx=0).stop 2024-01-16T13:53:41.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 msgr2=0x7fbe78150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 0x7fbe78150c40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbe680090b0 tx=0x7fbe68004750 comp rx=0 tx=0).stop 2024-01-16T13:53:41.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 shutdown_connections 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fbe5806b3e0 0x7fbe5806d8a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe7815b240 0x7fbe7809cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe7806c2e0 0x7fbe78150c40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 --2- 172.21.15.113:0/4276070779 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe7806e9a0 0x7fbe781f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 >> 172.21.15.113:0/4276070779 conn(0x7fbe780d0c00 msgr2=0x7fbe780d11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 shutdown_connections 2024-01-16T13:53:41.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.120+0000 7fbe7e8e4700 1 -- 172.21.15.113:0/4276070779 wait complete. 2024-01-16T13:53:41.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T13:53:41.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.261+0000 7f614343a700 1 Processor -- start 2024-01-16T13:53:41.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.261+0000 7f614343a700 1 -- start start 2024-01-16T13:53:41.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f614343a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f613c1446a0 0x7f613c0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f614343a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f613c0adf00 0x7f613c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f614343a700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c0af950 con 0x7f613c06c540 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f614343a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613c06c9f0 con 0x7f613c1446a0 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f614343a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613c144a80 con 0x7f613c0adf00 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61411d6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f613c1446a0 0x7f613c0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61419d7700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f613c06c540 0x7f613c06da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:36414/0 (socket says 172.21.15.113:36414) 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61411d6700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f613c1446a0 0x7f613c0b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37910/0 (socket says 172.21.15.113:37910) 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61419d7700 1 -- 172.21.15.113:0/3526096505 learned_addr learned my addr 172.21.15.113:0/3526096505 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61409d5700 1 --2- 172.21.15.113:0/3526096505 >> v2:172.21.15.124:3300/0 conn(0x7f613c0adf00 0x7f613c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61411d6700 1 -- 172.21.15.113:0/3526096505 >> v2:172.21.15.124:3300/0 conn(0x7f613c0adf00 msgr2=0x7f613c06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61411d6700 1 --2- 172.21.15.113:0/3526096505 >> v2:172.21.15.124:3300/0 conn(0x7f613c0adf00 0x7f613c06c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61411d6700 1 -- 172.21.15.113:0/3526096505 >> v1:172.21.15.113:6789/0 conn(0x7f613c06c540 legacy=0x7f613c06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f61411d6700 1 -- 172.21.15.113:0/3526096505 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0afc30 con 0x7f613c1446a0 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.262+0000 7f612ffff700 1 -- 172.21.15.113:0/3526096505 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482468441 0 0) 0x7f613c0af950 con 0x7f613c06c540 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f61411d6700 1 --2- 172.21.15.113:0/3526096505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c1446a0 0x7f613c0b5ae0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f613c09f030 tx=0x7f6138009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce01699845ac9a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f612ffff700 1 -- 172.21.15.113:0/3526096505 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138011830 con 0x7f613c1446a0 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f612ffff700 1 -- 172.21.15.113:0/3526096505 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138011990 con 0x7f613c1446a0 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f612ffff700 1 -- 172.21.15.113:0/3526096505 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138011d20 con 0x7f613c1446a0 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 -- 172.21.15.113:0/3526096505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c1446a0 msgr2=0x7f613c0b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 --2- 172.21.15.113:0/3526096505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c1446a0 0x7f613c0b5ae0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f613c09f030 tx=0x7f6138009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 -- 172.21.15.113:0/3526096505 shutdown_connections 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 --2- 172.21.15.113:0/3526096505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c1446a0 0x7f613c0b5ae0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 --2- 172.21.15.113:0/3526096505 >> v2:172.21.15.124:3300/0 conn(0x7f613c0adf00 0x7f613c06c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 -- 172.21.15.113:0/3526096505 >> 172.21.15.113:0/3526096505 conn(0x7f613c071b90 msgr2=0x7f613c071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 -- 172.21.15.113:0/3526096505 shutdown_connections 2024-01-16T13:53:41.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.263+0000 7f614343a700 1 -- 172.21.15.113:0/3526096505 wait complete. 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.265+0000 7f614343a700 1 Processor -- start 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.265+0000 7f614343a700 1 -- start start 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f614343a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f613c06c540 0x7f613c1f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f614343a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 0x7f613c1408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f614343a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f613c1446a0 0x7f613c0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f614343a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c143790 con 0x7f613c0adf00 2024-01-16T13:53:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f614343a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c141f50 con 0x7f613c06c540 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f614343a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613c141340 con 0x7f613c1446a0 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 0x7f613c1408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 0x7f613c1408c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37918/0 (socket says 172.21.15.113:37918) 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 -- 172.21.15.113:0/3091618506 learned_addr learned my addr 172.21.15.113:0/3091618506 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61411d6700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f613c06c540 0x7f613c1f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61419d7700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f613c1446a0 0x7f613c0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 -- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f613c1446a0 msgr2=0x7f613c0b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f613c1446a0 0x7f613c0b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 -- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f613c06c540 msgr2=0x7f613c1f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f613c06c540 0x7f613c1f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 -- 172.21.15.113:0/3091618506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380094a0 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61419d7700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f613c1446a0 0x7f613c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.266+0000 7f61409d5700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 0x7f613c1408c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f613000e810 tx=0x7f613000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.267+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130013610 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.267+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6130013770 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.267+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c141620 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.267+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130013b20 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.267+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c141be0 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.268+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613c0753f0 con 0x7f613c0adf00 2024-01-16T13:53:41.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.268+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f6130013c80 con 0x7f613c0adf00 2024-01-16T13:53:41.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.269+0000 7f612dffb700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f612006b500 0x7f612006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.269+0000 7f61411d6700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f612006b500 0x7f612006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.269+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6130089ae0 con 0x7f613c0adf00 2024-01-16T13:53:41.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.269+0000 7f61411d6700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f612006b500 0x7f612006d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f61380090b0 tx=0x7f613801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:41.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.273+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61300545b0 con 0x7f613c0adf00 2024-01-16T13:53:41.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.540+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f613c0d0620 con 0x7f613c0adf00 2024-01-16T13:53:41.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.540+0000 7f612dffb700 1 -- 172.21.15.113:0/3091618506 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+3292 (secure 0 0 0) 0x7f6130021cf0 con 0x7f613c0adf00 2024-01-16T13:53:41.523 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:41.523 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:41.523 INFO:teuthology.orchestra.run.smithi113.stdout: "fsid": "e77ab909-7680-4e53-a08d-a26ae42f1a8c", 2024-01-16T13:53:41.523 INFO:teuthology.orchestra.run.smithi113.stdout: "health": { 2024-01-16T13:53:41.523 INFO:teuthology.orchestra.run.smithi113.stdout: "status": "HEALTH_OK", 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "checks": {}, 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "mutes": [] 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "election_epoch": 4, 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "quorum": [ 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: 0, 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: 1, 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: 2 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "quorum_names": [ 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "a", 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "b", 2024-01-16T13:53:41.524 INFO:teuthology.orchestra.run.smithi113.stdout: "c" 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "quorum_age": 114, 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "monmap": { 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 1, 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "min_mon_release_name": "reef", 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "num_mons": 3 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap": { 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 33, 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "num_osds": 8, 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "num_up_osds": 8, 2024-01-16T13:53:41.525 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_up_since": 1705413116, 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "num_in_osds": 8, 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_in_since": 1705413109, 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "num_remapped_pgs": 0 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap": { 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "pgs_by_state": [ 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "state_name": "active+clean", 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "count": 97 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "num_pgs": 97, 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "num_pools": 3, 2024-01-16T13:53:41.526 INFO:teuthology.orchestra.run.smithi113.stdout: "num_objects": 796, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "data_bytes": 3222058787, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "bytes_used": 6668435456, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "bytes_avail": 766425677824, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "bytes_total": 773094113280, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "read_bytes_sec": 85, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "write_bytes_sec": 89465220, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "read_op_per_sec": 0, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "write_op_per_sec": 23 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "fsmap": { 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 17, 2024-01-16T13:53:41.527 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 2, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "up": 1, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "in": 1, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "by_rank": [ 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "filesystem_id": 2, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "rank": 0, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "d", 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "status": "up:active", 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 5067 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "up:standby": 3 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.528 INFO:teuthology.orchestra.run.smithi113.stdout: "mgrmap": { 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "available": true, 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "num_standbys": 1, 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "modules": [ 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "iostat", 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "nfs", 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "restful" 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "services": {} 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "servicemap": { 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "epoch": 4, 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "modified": "2024-01-16T13:52:58.412474+0000", 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "services": { 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-01-16T13:53:41.529 INFO:teuthology.orchestra.run.smithi113.stdout: "daemons": { 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "summary": "", 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "a": { 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "b": { 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T13:53:41.530 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "c": { 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "d": { 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "start_epoch": 0, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "start_stamp": "0.000000", 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "gid": 0, 2024-01-16T13:53:41.531 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": {}, 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: "task_status": {} 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_events": {} 2024-01-16T13:53:41.532 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:41.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f612006b500 msgr2=0x7f612006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f612006b500 0x7f612006d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f61380090b0 tx=0x7f613801c040 comp rx=0 tx=0).stop 2024-01-16T13:53:41.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 msgr2=0x7f613c1408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 0x7f613c1408c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f613000e810 tx=0x7f613000eb20 comp rx=0 tx=0).stop 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 shutdown_connections 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f612006b500 0x7f612006d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f613c1446a0 0x7f613c0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f613c06c540 0x7f613c1f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 --2- 172.21.15.113:0/3091618506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f613c0adf00 0x7f613c1408c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 >> 172.21.15.113:0/3091618506 conn(0x7f613c071b90 msgr2=0x7f613c0b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 shutdown_connections 2024-01-16T13:53:41.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.542+0000 7f614343a700 1 -- 172.21.15.113:0/3091618506 wait complete. 2024-01-16T13:53:41.537 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T13:53:41.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:41.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.679+0000 7f5ed19c9700 1 Processor -- start 2024-01-16T13:53:41.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.679+0000 7f5ed19c9700 1 -- start start 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5ed19c9700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5ecc161510 0x7f5ecc0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5ed19c9700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ecc0b98e0 0x7f5ecc0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5ed19c9700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc162af0 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5ed19c9700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc0d13e0 con 0x7f5ecc0b98e0 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5ed19c9700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc161f70 con 0x7f5ecc161510 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5eca7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ecc0b98e0 0x7f5ecc0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5eca7fc700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5ecc0b98e0 0x7f5ecc0cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37930/0 (socket says 172.21.15.113:37930) 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5eca7fc700 1 -- 172.21.15.113:0/115099383 learned_addr learned my addr 172.21.15.113:0/115099383 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5eca7fc700 1 -- 172.21.15.113:0/115099383 >> v2:172.21.15.124:3300/0 conn(0x7f5ecc161510 msgr2=0x7f5ecc0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5ecaffd700 1 --2- 172.21.15.113:0/115099383 >> v2:172.21.15.124:3300/0 conn(0x7f5ecc161510 0x7f5ecc0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5eca7fc700 1 --2- 172.21.15.113:0/115099383 >> v2:172.21.15.124:3300/0 conn(0x7f5ecc161510 0x7f5ecc0cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.680+0000 7f5eca7fc700 1 -- 172.21.15.113:0/115099383 >> v1:172.21.15.124:6789/0 conn(0x7f5ecc0b59d0 legacy=0x7f5ecc0b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5eca7fc700 1 -- 172.21.15.113:0/115099383 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc162250 con 0x7f5ecc0b98e0 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5eca7fc700 1 --2- 172.21.15.113:0/115099383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cf160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0d49f0 tx=0x7f5ebc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5a2ab2e1ecee6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ec9ffb700 1 -- 172.21.15.113:0/115099383 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc004230 con 0x7f5ecc0b98e0 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ec9ffb700 1 -- 172.21.15.113:0/115099383 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc004390 con 0x7f5ecc0b98e0 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ec9ffb700 1 -- 172.21.15.113:0/115099383 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0036a0 con 0x7f5ecc0b98e0 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 -- 172.21.15.113:0/115099383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b98e0 msgr2=0x7f5ecc0cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/115099383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cf160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0d49f0 tx=0x7f5ebc009bd0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 -- 172.21.15.113:0/115099383 shutdown_connections 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/115099383 >> v2:172.21.15.124:3300/0 conn(0x7f5ecc161510 0x7f5ecc0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/115099383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cf160 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 -- 172.21.15.113:0/115099383 >> 172.21.15.113:0/115099383 conn(0x7f5ecc06b410 msgr2=0x7f5ecc06b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 -- 172.21.15.113:0/115099383 shutdown_connections 2024-01-16T13:53:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.681+0000 7f5ed19c9700 1 -- 172.21.15.113:0/115099383 wait complete. 2024-01-16T13:53:41.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.683+0000 7f5ed19c9700 1 Processor -- start 2024-01-16T13:53:41.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.683+0000 7f5ed19c9700 1 -- start start 2024-01-16T13:53:41.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.683+0000 7f5ed19c9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 0x7f5ecc14e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.683+0000 7f5ed19c9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ed19c9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ecc161510 0x7f5ecc1f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ed19c9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0cc0a0 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ed19c9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc1f5240 con 0x7f5ecc0b98e0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ed19c9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0cbae0 con 0x7f5ecc161510 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 0x7f5ecc14e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 0x7f5ecc14e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37940/0 (socket says 172.21.15.113:37940) 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 -- 172.21.15.113:0/1809324976 learned_addr learned my addr 172.21.15.113:0/1809324976 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5eca7fc700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ecc161510 0x7f5ecc1f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 -- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ecc161510 msgr2=0x7f5ecc1f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ecc161510 0x7f5ecc1f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 -- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ecc0b98e0 msgr2=0x7f5ecc0cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 -- 172.21.15.113:0/1809324976 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc009550 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecb7fe700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ecc161510 0x7f5ecc1f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ecaffd700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 0x7f5ecc14e470 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000c7f0 tx=0x7f5ec000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011800 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0011960 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc0cbdc0 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011cf0 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.684+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc1f7f40 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.685+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc1f4d10 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.686+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f5ec0018360 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.686+0000 7f5eb3fff700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5eb806b500 0x7f5eb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.686+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ec0089f30 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.686+0000 7f5eca7fc700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5eb806b500 0x7f5eb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:41.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.686+0000 7f5eca7fc700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5eb806b500 0x7f5eb806d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0d49f0 tx=0x7f5ebc008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.691+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ec0054a20 con 0x7f5ecc0b59d0 2024-01-16T13:53:41.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.921+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ecc1618f0 con 0x7f5eb806b500 2024-01-16T13:53:41.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.927+0000 7f5eb3fff700 1 -- 172.21.15.113:0/1809324976 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f5ebc0041b0 con 0x7f5eb806b500 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 0.0, 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "0 B/3 GB", 2024-01-16T13:53:41.910 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "0/3", 2024-01-16T13:53:41.911 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:41.911 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:41.911 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.911 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:41.911 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:41.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5eb806b500 msgr2=0x7f5eb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5eb806b500 0x7f5eb806d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0d49f0 tx=0x7f5ebc008030 comp rx=0 tx=0).stop 2024-01-16T13:53:41.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 msgr2=0x7f5ecc14e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:41.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 0x7f5ecc14e470 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000c7f0 tx=0x7f5ec000cbb0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 shutdown_connections 2024-01-16T13:53:41.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f5eb806b500 0x7f5eb806d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ecc161510 0x7f5ecc1f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ecc0b59d0 0x7f5ecc14e470 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 --2- 172.21.15.113:0/1809324976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ecc0b98e0 0x7f5ecc0cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:41.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 >> 172.21.15.113:0/1809324976 conn(0x7f5ecc06b410 msgr2=0x7f5ecc0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:41.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 shutdown_connections 2024-01-16T13:53:41.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:41.929+0000 7f5ed19c9700 1 -- 172.21.15.113:0/1809324976 wait complete. 2024-01-16T13:53:41.924 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:42.925 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:43.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.081+0000 7f14748c7700 1 Processor -- start 2024-01-16T13:53:43.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.081+0000 7f14748c7700 1 -- start start 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f14748c7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700afb80 0x7f1470073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f14748c7700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f14700744d0 0x7f14700add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f14748c7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147015e4a0 con 0x7f147015e0c0 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f14748c7700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1470074960 con 0x7f14700744d0 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f14748c7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700affe0 con 0x7f14700afb80 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f146ed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f14700744d0 0x7f14700add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f146e59c700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f147015e0c0 0x7f14700b3bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:36418/0 (socket says 172.21.15.113:36418) 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f146ed9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f14700744d0 0x7f14700add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37948/0 (socket says 172.21.15.113:37948) 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.082+0000 7f146e59c700 1 -- 172.21.15.113:0/3291194510 learned_addr learned my addr 172.21.15.113:0/3291194510 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146dd9b700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700afb80 0x7f1470073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146ed9d700 1 -- 172.21.15.113:0/3291194510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700afb80 msgr2=0x7f1470073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:43.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146ed9d700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700afb80 0x7f1470073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146ed9d700 1 -- 172.21.15.113:0/3291194510 >> v1:172.21.15.113:6789/0 conn(0x7f147015e0c0 legacy=0x7f14700b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146ed9d700 1 -- 172.21.15.113:0/3291194510 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147006aff0 con 0x7f14700744d0 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146d59a700 1 -- 172.21.15.113:0/3291194510 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168272643 0 0) 0x7f147015e4a0 con 0x7f147015e0c0 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146dd9b700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700afb80 0x7f1470073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146ed9d700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14700744d0 0x7f14700add40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1468009d90 tx=0x7f146800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fc86fce01386d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146d59a700 1 -- 172.21.15.113:0/3291194510 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146800e070 con 0x7f14700744d0 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146d59a700 1 -- 172.21.15.113:0/3291194510 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146800bbd0 con 0x7f14700744d0 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.083+0000 7f146d59a700 1 -- 172.21.15.113:0/3291194510 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468012430 con 0x7f14700744d0 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 -- 172.21.15.113:0/3291194510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14700744d0 msgr2=0x7f14700add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14700744d0 0x7f14700add40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1468009d90 tx=0x7f146800b330 comp rx=0 tx=0).stop 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 -- 172.21.15.113:0/3291194510 shutdown_connections 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14700744d0 0x7f14700add40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 --2- 172.21.15.113:0/3291194510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700afb80 0x7f1470073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 -- 172.21.15.113:0/3291194510 >> 172.21.15.113:0/3291194510 conn(0x7f14700d0c00 msgr2=0x7f14700d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 -- 172.21.15.113:0/3291194510 shutdown_connections 2024-01-16T13:53:43.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.084+0000 7f14748c7700 1 -- 172.21.15.113:0/3291194510 wait complete. 2024-01-16T13:53:43.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.085+0000 7f14748c7700 1 Processor -- start 2024-01-16T13:53:43.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.085+0000 7f14748c7700 1 -- start start 2024-01-16T13:53:43.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f14748c7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700744d0 0x7f147013f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:43.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f14748c7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f14700afb80 0x7f147015e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:43.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f14748c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 0x7f147015ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:43.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f14748c7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470142b20 con 0x7f147015e0c0 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f14748c7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701421f0 con 0x7f14700afb80 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f14748c7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14701424d0 con 0x7f14700744d0 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 0x7f147015ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 0x7f147015ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37950/0 (socket says 172.21.15.113:37950) 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 -- 172.21.15.113:0/3487221101 learned_addr learned my addr 172.21.15.113:0/3487221101 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146e59c700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700744d0 0x7f147013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146dd9b700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f14700afb80 0x7f147015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 -- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700744d0 msgr2=0x7f147013f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700744d0 0x7f147013f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 -- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f14700afb80 msgr2=0x7f147015e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f14700afb80 0x7f147015e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146ed9d700 1 -- 172.21.15.113:0/3487221101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1468009a70 con 0x7f147015e0c0 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146e59c700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700744d0 0x7f147013f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.086+0000 7f146dd9b700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f14700afb80 0x7f147015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:43.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.087+0000 7f146ed9d700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 0x7f147015ebd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f147006db90 tx=0x7f1468004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.087+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468012430 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.087+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468020070 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.087+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701427b0 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.087+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468004280 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.087+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701f47b0 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.088+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147006c090 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.088+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f14680043e0 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.088+0000 7f145effd700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f145006b500 0x7f145006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.089+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1468022070 con 0x7f147015e0c0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.089+0000 7f146e59c700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f145006b500 0x7f145006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:43.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.089+0000 7f146e59c700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f145006b500 0x7f145006d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1460000f60 tx=0x7f1460009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:43.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.093+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1468056a20 con 0x7f147015e0c0 2024-01-16T13:53:43.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.407+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1470142360 con 0x7f145006b500 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.423+0000 7f145effd700 1 -- 172.21.15.113:0/3487221101 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f1470142360 con 0x7f145006b500 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 0.0, 2024-01-16T13:53:43.405 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "0 B/3 GB", 2024-01-16T13:53:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "0/3", 2024-01-16T13:53:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:43.406 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:43.406 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:43.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f145006b500 msgr2=0x7f145006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:43.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f145006b500 0x7f145006d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1460000f60 tx=0x7f1460009040 comp rx=0 tx=0).stop 2024-01-16T13:53:43.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 msgr2=0x7f147015ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:43.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 0x7f147015ebd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f147006db90 tx=0x7f1468004ad0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 shutdown_connections 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f145006b500 0x7f145006d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f147015e0c0 0x7f147015ebd0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f14700744d0 0x7f147013f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 --2- 172.21.15.113:0/3487221101 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f14700afb80 0x7f147015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 >> 172.21.15.113:0/3487221101 conn(0x7f14700d0c00 msgr2=0x7f14700aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 shutdown_connections 2024-01-16T13:53:43.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:43.426+0000 7f14748c7700 1 -- 172.21.15.113:0/3487221101 wait complete. 2024-01-16T13:53:43.433 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:44.434 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:44.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.692+0000 7f26814c0700 1 Processor -- start 2024-01-16T13:53:44.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.692+0000 7f26814c0700 1 -- start start 2024-01-16T13:53:44.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f26814c0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15abf0 0x7f267c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:44.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f26814c0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f267c15efb0 0x7f267c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:44.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f26814c0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c06d460 con 0x7f267c072010 2024-01-16T13:53:44.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f26814c0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f267c0b4790 con 0x7f267c15efb0 2024-01-16T13:53:44.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f26814c0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267c0b4a70 con 0x7f267c15abf0 2024-01-16T13:53:44.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f267b7fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f267c15efb0 0x7f267c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:44.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f267affd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15abf0 0x7f267c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:44.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f267b7fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f267c15efb0 0x7f267c06d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:59034/0 (socket says 172.21.15.113:59034) 2024-01-16T13:53:44.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.695+0000 7f267b7fe700 1 -- 172.21.15.113:0/2265751258 learned_addr learned my addr 172.21.15.113:0/2265751258 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.696+0000 7f267b7fe700 1 -- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15abf0 msgr2=0x7f267c0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.696+0000 7f267b7fe700 1 --2- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15abf0 0x7f267c0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.696+0000 7f267b7fe700 1 -- 172.21.15.113:0/2265751258 >> v1:172.21.15.124:6789/0 conn(0x7f267c072010 legacy=0x7f267c15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.696+0000 7f267b7fe700 1 -- 172.21.15.113:0/2265751258 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c0b4d50 con 0x7f267c15efb0 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.696+0000 7f267b7fe700 1 --2- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15efb0 0x7f267c06d8f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2674009e50 tx=0x7f267400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a5e2761ad5f9990 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.696+0000 7f2679ffb700 1 -- 172.21.15.113:0/2265751258 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267400e070 con 0x7f267c15efb0 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.697+0000 7f2679ffb700 1 -- 172.21.15.113:0/2265751258 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2674003a20 con 0x7f267c15efb0 2024-01-16T13:53:44.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.697+0000 7f2679ffb700 1 -- 172.21.15.113:0/2265751258 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674003dd0 con 0x7f267c15efb0 2024-01-16T13:53:44.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.698+0000 7f26814c0700 1 -- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15efb0 msgr2=0x7f267c06d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:44.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.698+0000 7f26814c0700 1 --2- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15efb0 0x7f267c06d8f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2674009e50 tx=0x7f267400b330 comp rx=0 tx=0).stop 2024-01-16T13:53:44.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.699+0000 7f26814c0700 1 -- 172.21.15.113:0/2265751258 shutdown_connections 2024-01-16T13:53:44.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.699+0000 7f26814c0700 1 --2- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15efb0 0x7f267c06d8f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.699+0000 7f26814c0700 1 --2- 172.21.15.113:0/2265751258 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15abf0 0x7f267c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.699+0000 7f26814c0700 1 -- 172.21.15.113:0/2265751258 >> 172.21.15.113:0/2265751258 conn(0x7f267c0743b0 msgr2=0x7f267c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:44.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.701+0000 7f26814c0700 1 -- 172.21.15.113:0/2265751258 shutdown_connections 2024-01-16T13:53:44.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.701+0000 7f26814c0700 1 -- 172.21.15.113:0/2265751258 wait complete. 2024-01-16T13:53:44.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.705+0000 7f26814c0700 1 Processor -- start 2024-01-16T13:53:44.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.705+0000 7f26814c0700 1 -- start start 2024-01-16T13:53:44.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.708+0000 7f26814c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f267c072010 0x7f267c14c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:44.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.708+0000 7f26814c0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15abf0 0x7f267c1545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:44.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.708+0000 7f26814c0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15efb0 0x7f267c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:44.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.708+0000 7f26814c0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c15dd50 con 0x7f267c072010 2024-01-16T13:53:44.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.708+0000 7f26814c0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c157970 con 0x7f267c15abf0 2024-01-16T13:53:44.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.708+0000 7f26814c0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267c15d790 con 0x7f267c15efb0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.709+0000 7f267a7fc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15abf0 0x7f267c1545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.709+0000 7f267a7fc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15abf0 0x7f267c1545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:59040/0 (socket says 172.21.15.113:59040) 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.709+0000 7f267a7fc700 1 -- 172.21.15.113:0/1581444213 learned_addr learned my addr 172.21.15.113:0/1581444213 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.709+0000 7f267b7fe700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15efb0 0x7f267c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.710+0000 7f267b7fe700 1 -- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15abf0 msgr2=0x7f267c1545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.710+0000 7f267b7fe700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15abf0 0x7f267c1545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.710+0000 7f267b7fe700 1 -- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f267c072010 msgr2=0x7f267c14c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.710+0000 7f267b7fe700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f267c072010 0x7f267c14c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.710+0000 7f267b7fe700 1 -- 172.21.15.113:0/1581444213 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2674009b30 con 0x7f267c15efb0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.710+0000 7f267b7fe700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15efb0 0x7f267c14fff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2674014000 tx=0x7f2674004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.712+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674004440 con 0x7f267c15efb0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.712+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26740045a0 con 0x7f267c15efb0 2024-01-16T13:53:44.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.712+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c15da70 con 0x7f267c15efb0 2024-01-16T13:53:44.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.712+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674004950 con 0x7f267c15efb0 2024-01-16T13:53:44.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.712+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c152220 con 0x7f267c15efb0 2024-01-16T13:53:44.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.714+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f2674019050 con 0x7f267c15efb0 2024-01-16T13:53:44.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.714+0000 7f26637fe700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f266806b550 0x7f266806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:44.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.715+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f267408ae50 con 0x7f267c15efb0 2024-01-16T13:53:44.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.716+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c0052f0 con 0x7f267c15efb0 2024-01-16T13:53:44.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.718+0000 7f267affd700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f266806b550 0x7f266806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:44.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.719+0000 7f267affd700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f266806b550 0x7f266806da10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f266c006b20 tx=0x7f266c006ab0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:44.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:44.725+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2674055ae0 con 0x7f267c15efb0 2024-01-16T13:53:44.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.000+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f265c000bc0 con 0x7f266806b550 2024-01-16T13:53:44.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.006+0000 7f26637fe700 1 -- 172.21.15.113:0/1581444213 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f265c000bc0 con 0x7f266806b550 2024-01-16T13:53:44.987 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 33.33333333333333, 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "1/3", 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:44.988 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:44.989 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:44.989 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f266806b550 msgr2=0x7f266806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f266806b550 0x7f266806da10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f266c006b20 tx=0x7f266c006ab0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15efb0 msgr2=0x7f267c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15efb0 0x7f267c14fff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2674014000 tx=0x7f2674004ad0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 shutdown_connections 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f266806b550 0x7f266806da10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f267c15efb0 0x7f267c14fff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f267c072010 0x7f267c14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 --2- 172.21.15.113:0/1581444213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f267c15abf0 0x7f267c1545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 >> 172.21.15.113:0/1581444213 conn(0x7f267c0743b0 msgr2=0x7f267c0b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 shutdown_connections 2024-01-16T13:53:44.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:45.008+0000 7f26814c0700 1 -- 172.21.15.113:0/1581444213 wait complete. 2024-01-16T13:53:45.001 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:46.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:46.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.160+0000 7f1837c09700 1 Processor -- start 2024-01-16T13:53:46.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.161+0000 7f1837c09700 1 -- start start 2024-01-16T13:53:46.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.161+0000 7f1837c09700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f183015dee0 0x7f183006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:46.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.161+0000 7f1837c09700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f18300d2350 0x7f18300d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:46.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.161+0000 7f1837c09700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183006b840 con 0x7f183015e2c0 2024-01-16T13:53:46.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.161+0000 7f1837c09700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183006c3c0 con 0x7f183015dee0 2024-01-16T13:53:46.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.161+0000 7f1837c09700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183006bb20 con 0x7f18300d2350 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f183015dee0 0x7f183006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f183015dee0 0x7f183006aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36776/0 (socket says 172.21.15.113:36776) 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 -- 172.21.15.113:0/2164624252 learned_addr learned my addr 172.21.15.113:0/2164624252 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18361a6700 1 --2- 172.21.15.113:0/2164624252 >> v2:172.21.15.124:3300/0 conn(0x7f18300d2350 0x7f18300d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 -- 172.21.15.113:0/2164624252 >> v2:172.21.15.124:3300/0 conn(0x7f18300d2350 msgr2=0x7f18300d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 --2- 172.21.15.113:0/2164624252 >> v2:172.21.15.124:3300/0 conn(0x7f18300d2350 0x7f18300d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 -- 172.21.15.113:0/2164624252 >> v1:172.21.15.124:6789/0 conn(0x7f183015e2c0 legacy=0x7f18300d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 -- 172.21.15.113:0/2164624252 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183006be00 con 0x7f183015dee0 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18359a5700 1 --2- 172.21.15.113:0/2164624252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f183006aff0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f183006cd10 tx=0x7f1820009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09f4bc2f1d42643 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18349a3700 1 -- 172.21.15.113:0/2164624252 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820004230 con 0x7f183015dee0 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18349a3700 1 -- 172.21.15.113:0/2164624252 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1820004390 con 0x7f183015dee0 2024-01-16T13:53:46.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.162+0000 7f18349a3700 1 -- 172.21.15.113:0/2164624252 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200036a0 con 0x7f183015dee0 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 -- 172.21.15.113:0/2164624252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 msgr2=0x7f183006aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 --2- 172.21.15.113:0/2164624252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f183006aff0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f183006cd10 tx=0x7f1820009bd0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 -- 172.21.15.113:0/2164624252 shutdown_connections 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 --2- 172.21.15.113:0/2164624252 >> v2:172.21.15.124:3300/0 conn(0x7f18300d2350 0x7f18300d0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 --2- 172.21.15.113:0/2164624252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f183006aff0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 -- 172.21.15.113:0/2164624252 >> 172.21.15.113:0/2164624252 conn(0x7f1830073130 msgr2=0x7f183009aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 -- 172.21.15.113:0/2164624252 shutdown_connections 2024-01-16T13:53:46.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.163+0000 7f1837c09700 1 -- 172.21.15.113:0/2164624252 wait complete. 2024-01-16T13:53:46.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.164+0000 7f1837c09700 1 Processor -- start 2024-01-16T13:53:46.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.164+0000 7f1837c09700 1 -- start start 2024-01-16T13:53:46.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18300d2350 0x7f18301f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:46.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f18300b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:46.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f183015e2c0 0x7f18300cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:46.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18301f36b0 con 0x7f183015dee0 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183015f100 con 0x7f18300d2350 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18300b5040 con 0x7f183015e2c0 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f18300b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f18300b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36788/0 (socket says 172.21.15.113:36788) 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 -- 172.21.15.113:0/1686136058 learned_addr learned my addr 172.21.15.113:0/1686136058 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18361a6700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f183015e2c0 0x7f18300cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18359a5700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18300d2350 0x7f18301f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 -- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f183015e2c0 msgr2=0x7f18300cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f183015e2c0 0x7f18300cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 -- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18300d2350 msgr2=0x7f18301f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18300d2350 0x7f18301f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 -- 172.21.15.113:0/1686136058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820009550 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18351a4700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f18300b4b00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f182c00b450 tx=0x7f182c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c011830 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c011990 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18300b5310 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.165+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c011d20 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.166+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18300b58d0 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.166+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830068450 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.167+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f182c01a400 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.167+0000 7f18267fc700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f181406b500 0x7f181406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.167+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f182c089ef0 con 0x7f183015dee0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.167+0000 7f18359a5700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f181406b500 0x7f181406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:46.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.168+0000 7f18359a5700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f181406b500 0x7f181406d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1820000c00 tx=0x7f1820008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:46.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.172+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f182c054b70 con 0x7f183015dee0 2024-01-16T13:53:46.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.405+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f183006b530 con 0x7f181406b500 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.411+0000 7f18267fc700 1 -- 172.21.15.113:0/1686136058 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f1820020070 con 0x7f181406b500 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 33.33333333333333, 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "1/3", 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:46.394 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f181406b500 msgr2=0x7f181406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f181406b500 0x7f181406d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1820000c00 tx=0x7f1820008030 comp rx=0 tx=0).stop 2024-01-16T13:53:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 msgr2=0x7f18300b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f18300b4b00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f182c00b450 tx=0x7f182c00b810 comp rx=0 tx=0).stop 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 shutdown_connections 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f181406b500 0x7f181406d9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f183015e2c0 0x7f18300cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18300d2350 0x7f18301f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 --2- 172.21.15.113:0/1686136058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f183015dee0 0x7f18300b4b00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 >> 172.21.15.113:0/1686136058 conn(0x7f1830073130 msgr2=0x7f18300b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 shutdown_connections 2024-01-16T13:53:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:46.413+0000 7f1837c09700 1 -- 172.21.15.113:0/1686136058 wait complete. 2024-01-16T13:53:46.407 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:47.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:47.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.639+0000 7f9f9f90c700 1 Processor -- start 2024-01-16T13:53:47.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.640+0000 7f9f9f90c700 1 -- start start 2024-01-16T13:53:47.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9f90c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9f98161510 0x7f9f980cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:47.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9f90c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 0x7f9f980cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:47.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9f90c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f98162af0 con 0x7f9f980b59d0 2024-01-16T13:53:47.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9f90c700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f980d13e0 con 0x7f9f98161510 2024-01-16T13:53:47.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9f90c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f98161f70 con 0x7f9f980b98e0 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 0x7f9f980cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 0x7f9f980cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:57940/0 (socket says 172.21.15.113:57940) 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3982138580 learned_addr learned my addr 172.21.15.113:0/3982138580 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9dea9700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f9f980b59d0 0x7f9f980b5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:52958/0 (socket says 172.21.15.113:52958) 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9d6a8700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9f98161510 0x7f9f980cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3982138580 >> v2:172.21.15.124:3300/0 conn(0x7f9f98161510 msgr2=0x7f9f980cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 --2- 172.21.15.113:0/3982138580 >> v2:172.21.15.124:3300/0 conn(0x7f9f98161510 0x7f9f980cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3982138580 >> v1:172.21.15.124:6789/0 conn(0x7f9f980b59d0 legacy=0x7f9f980b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.641+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3982138580 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f98162250 con 0x7f9f980b98e0 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.642+0000 7f9f9d6a8700 1 --2- 172.21.15.113:0/3982138580 >> v2:172.21.15.124:3300/0 conn(0x7f9f98161510 0x7f9f980cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.642+0000 7f9f9cea7700 1 --2- 172.21.15.113:0/3982138580 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 0x7f9f980cf160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f880098f0 tx=0x7f9f88009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16a009a7ef341b6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.643+0000 7f9f8ffff700 1 -- 172.21.15.113:0/3982138580 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88011830 con 0x7f9f980b98e0 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.643+0000 7f9f8ffff700 1 -- 172.21.15.113:0/3982138580 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f88011990 con 0x7f9f980b98e0 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.643+0000 7f9f8ffff700 1 -- 172.21.15.113:0/3982138580 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88011d20 con 0x7f9f980b98e0 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.643+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3982138580 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 msgr2=0x7f9f980cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.643+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3982138580 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 0x7f9f980cf160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f880098f0 tx=0x7f9f88009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.644+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3982138580 shutdown_connections 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.644+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3982138580 >> v2:172.21.15.124:3300/0 conn(0x7f9f98161510 0x7f9f980cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.644+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3982138580 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f980b98e0 0x7f9f980cf160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.644+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3982138580 >> 172.21.15.113:0/3982138580 conn(0x7f9f9806b410 msgr2=0x7f9f9806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:47.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.644+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3982138580 shutdown_connections 2024-01-16T13:53:47.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.644+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3982138580 wait complete. 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.645+0000 7f9f9f90c700 1 Processor -- start 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.645+0000 7f9f9f90c700 1 -- start start 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9f90c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f980b59d0 0x7f9f980cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9f90c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 0x7f9f98145a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9f90c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f98161510 0x7f9f98071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9f90c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f98145f60 con 0x7f9f980b98e0 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9f90c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9815a1a0 con 0x7f9f980b59d0 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9f90c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9815a480 con 0x7f9f98161510 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 0x7f9f98145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:47.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9d6a8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f980b59d0 0x7f9f980cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9dea9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f98161510 0x7f9f98071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 0x7f9f98145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36798/0 (socket says 172.21.15.113:36798) 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3648377649 learned_addr learned my addr 172.21.15.113:0/3648377649 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f98161510 msgr2=0x7f9f98071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f98161510 0x7f9f98071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f980b59d0 msgr2=0x7f9f980cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f980b59d0 0x7f9f980cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 -- 172.21.15.113:0/3648377649 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f940094a0 con 0x7f9f980b98e0 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9d6a8700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f980b59d0 0x7f9f980cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f9cea7700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 0x7f9f98145a20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009c40 tx=0x7f9f88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88004550 con 0x7f9f980b98e0 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.646+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f880036a0 con 0x7f9f980b98e0 2024-01-16T13:53:47.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.647+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f88009550 con 0x7f9f980b98e0 2024-01-16T13:53:47.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.647+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88003a30 con 0x7f9f980b98e0 2024-01-16T13:53:47.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.647+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9815aa40 con 0x7f9f980b98e0 2024-01-16T13:53:47.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.648+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f980d3d50 con 0x7f9f980b98e0 2024-01-16T13:53:47.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.648+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f9f88021070 con 0x7f9f980b98e0 2024-01-16T13:53:47.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.649+0000 7f9f8dffb700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:47.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.649+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f8808b420 con 0x7f9f980b98e0 2024-01-16T13:53:47.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.653+0000 7f9f9d6a8700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:47.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.654+0000 7f9f9d6a8700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9f94011380 tx=0x7f9f94009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:47.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:47.655+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f88056140 con 0x7f9f980b98e0 2024-01-16T13:53:48.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.040+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f981618f0 con 0x7f9f7c06b500 2024-01-16T13:53:48.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.053+0000 7f9f8dffb700 1 -- 172.21.15.113:0/3648377649 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f9f94011d40 con 0x7f9f7c06b500 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 33.33333333333333, 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "1/3", 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:48.035 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:48.036 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:48.036 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:48.036 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:48.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9f7c06b500 msgr2=0x7f9f7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:48.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9f94011380 tx=0x7f9f94009210 comp rx=0 tx=0).stop 2024-01-16T13:53:48.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 msgr2=0x7f9f98145a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:48.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 0x7f9f98145a20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009c40 tx=0x7f9f88004750 comp rx=0 tx=0).stop 2024-01-16T13:53:48.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 shutdown_connections 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f9f7c06b500 0x7f9f7c06d9c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f98161510 0x7f9f98071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f980b59d0 0x7f9f980cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 --2- 172.21.15.113:0/3648377649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f980b98e0 0x7f9f98145a20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 >> 172.21.15.113:0/3648377649 conn(0x7f9f9806b410 msgr2=0x7f9f980b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 shutdown_connections 2024-01-16T13:53:48.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:48.055+0000 7f9f9f90c700 1 -- 172.21.15.113:0/3648377649 wait complete. 2024-01-16T13:53:48.062 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:49.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:49.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.209+0000 7f6578b02700 1 Processor -- start 2024-01-16T13:53:49.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.209+0000 7f6578b02700 1 -- start start 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.210+0000 7f6578b02700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f657415e0c0 0x7f65740afb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.210+0000 7f6578b02700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65740746a0 0x7f65740add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.210+0000 7f6578b02700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657415e4a0 con 0x7f65740b3bf0 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.210+0000 7f6578b02700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657406aff0 con 0x7f65740746a0 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.210+0000 7f6578b02700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657406b2d0 con 0x7f657415e0c0 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65740746a0 0x7f65740add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:49.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65740746a0 0x7f65740add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36800/0 (socket says 172.21.15.113:36800) 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 -- 172.21.15.113:0/62299074 learned_addr learned my addr 172.21.15.113:0/62299074 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f657259c700 1 --2- 172.21.15.113:0/62299074 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f657415e0c0 0x7f65740afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 -- 172.21.15.113:0/62299074 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f657415e0c0 msgr2=0x7f65740afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 --2- 172.21.15.113:0/62299074 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f657415e0c0 0x7f65740afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f657159a700 1 -- 172.21.15.113:0/62299074 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745298026 0 0) 0x7f657415e4a0 con 0x7f65740b3bf0 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 -- 172.21.15.113:0/62299074 >> v1:172.21.15.124:6789/0 conn(0x7f65740b3bf0 legacy=0x7f6574073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 -- 172.21.15.113:0/62299074 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657406b5b0 con 0x7f65740746a0 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f6572d9d700 1 --2- 172.21.15.113:0/62299074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65740746a0 0x7f65740add40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f656c009e50 tx=0x7f656c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94ede4bec90f7d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f657159a700 1 -- 172.21.15.113:0/62299074 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c00e070 con 0x7f65740746a0 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f657159a700 1 -- 172.21.15.113:0/62299074 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c00b910 con 0x7f65740746a0 2024-01-16T13:53:49.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.211+0000 7f657159a700 1 -- 172.21.15.113:0/62299074 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c00bcc0 con 0x7f65740746a0 2024-01-16T13:53:49.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 -- 172.21.15.113:0/62299074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65740746a0 msgr2=0x7f65740add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:49.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 --2- 172.21.15.113:0/62299074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65740746a0 0x7f65740add40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f656c009e50 tx=0x7f656c00b330 comp rx=0 tx=0).stop 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 -- 172.21.15.113:0/62299074 shutdown_connections 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 --2- 172.21.15.113:0/62299074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65740746a0 0x7f65740add40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 --2- 172.21.15.113:0/62299074 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f657415e0c0 0x7f65740afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 -- 172.21.15.113:0/62299074 >> 172.21.15.113:0/62299074 conn(0x7f65740d0c00 msgr2=0x7f65740d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 -- 172.21.15.113:0/62299074 shutdown_connections 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.212+0000 7f6578b02700 1 -- 172.21.15.113:0/62299074 wait complete. 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.213+0000 7f6578b02700 1 Processor -- start 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.213+0000 7f6578b02700 1 -- start start 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6578b02700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f65740746a0 0x7f6574141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6578b02700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65740b3bf0 0x7f6574152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6578b02700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 0x7f65741525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6578b02700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65741537e0 con 0x7f657415e0c0 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6578b02700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6574151cd0 con 0x7f65740b3bf0 2024-01-16T13:53:49.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6578b02700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6574152bd0 con 0x7f65740746a0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6572d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 0x7f65741525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6572d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 0x7f65741525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36810/0 (socket says 172.21.15.113:36810) 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6572d9d700 1 -- 172.21.15.113:0/4150567585 learned_addr learned my addr 172.21.15.113:0/4150567585 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f6571d9b700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65740b3bf0 0x7f6574152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.214+0000 7f657259c700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f65740746a0 0x7f6574141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6572d9d700 1 -- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f65740746a0 msgr2=0x7f6574141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6572d9d700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f65740746a0 0x7f6574141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6572d9d700 1 -- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65740b3bf0 msgr2=0x7f6574152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6572d9d700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65740b3bf0 0x7f6574152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6572d9d700 1 -- 172.21.15.113:0/4150567585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c009b30 con 0x7f657415e0c0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6571d9b700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65740b3bf0 0x7f6574152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6572d9d700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 0x7f65741525a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f656c012000 tx=0x7f656c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c003ea0 con 0x7f657415e0c0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c0099b0 con 0x7f657415e0c0 2024-01-16T13:53:49.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c01c680 con 0x7f657415e0c0 2024-01-16T13:53:49.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6574152eb0 con 0x7f657415e0c0 2024-01-16T13:53:49.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.215+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6574153470 con 0x7f657415e0c0 2024-01-16T13:53:49.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.216+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6574153630 con 0x7f657415e0c0 2024-01-16T13:53:49.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.216+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f656c01c7e0 con 0x7f657415e0c0 2024-01-16T13:53:49.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.217+0000 7f6562ffd700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f655406b500 0x7f655406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:49.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.217+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f656c08bc40 con 0x7f657415e0c0 2024-01-16T13:53:49.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.217+0000 7f657259c700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f655406b500 0x7f655406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:49.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.217+0000 7f657259c700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f655406b500 0x7f655406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6564009ab0 tx=0x7f6564008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:49.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.221+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f656c0567b0 con 0x7f657415e0c0 2024-01-16T13:53:49.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.451+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f65740b4050 con 0x7f655406b500 2024-01-16T13:53:49.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.457+0000 7f6562ffd700 1 -- 172.21.15.113:0/4150567585 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f65640106e0 con 0x7f655406b500 2024-01-16T13:53:49.438 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:49.438 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:49.438 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 66.66666666666666, 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "2/3", 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:49.439 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.458+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f655406b500 msgr2=0x7f655406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f655406b500 0x7f655406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6564009ab0 tx=0x7f6564008040 comp rx=0 tx=0).stop 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 msgr2=0x7f65741525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 0x7f65741525a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f656c012000 tx=0x7f656c004ad0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 shutdown_connections 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f655406b500 0x7f655406d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f657415e0c0 0x7f65741525a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f65740746a0 0x7f6574141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 --2- 172.21.15.113:0/4150567585 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65740b3bf0 0x7f6574152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:49.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 >> 172.21.15.113:0/4150567585 conn(0x7f65740d0c00 msgr2=0x7f6574073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:49.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 shutdown_connections 2024-01-16T13:53:49.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:49.459+0000 7f6578b02700 1 -- 172.21.15.113:0/4150567585 wait complete. 2024-01-16T13:53:49.453 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:50.453 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:50.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.639+0000 7f2419d8d700 1 Processor -- start 2024-01-16T13:53:50.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.639+0000 7f2419d8d700 1 -- start start 2024-01-16T13:53:50.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.640+0000 7f2419d8d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f24140b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:50.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.640+0000 7f2419d8d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f24140b2ee0 0x7f24140724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:50.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.640+0000 7f2419d8d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140b4fd0 con 0x7f24140b2a60 2024-01-16T13:53:50.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.640+0000 7f2419d8d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241413e010 con 0x7f24140b2ee0 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.640+0000 7f2419d8d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24140b4730 con 0x7f241415b320 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2412ffd700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f24140b2a60 0x7f24141409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:52976/0 (socket says 172.21.15.113:52976) 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2412ffd700 1 -- 172.21.15.113:0/398865853 learned_addr learned my addr 172.21.15.113:0/398865853 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f24137fe700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f24140b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2413fff700 1 --2- 172.21.15.113:0/398865853 >> v2:172.21.15.124:3300/0 conn(0x7f24140b2ee0 0x7f24140724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2413fff700 1 -- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 msgr2=0x7f24140b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2413fff700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f24140b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2413fff700 1 -- 172.21.15.113:0/398865853 >> v1:172.21.15.124:6789/0 conn(0x7f24140b2a60 legacy=0x7f24141409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:50.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f2413fff700 1 -- 172.21.15.113:0/398865853 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140b4a10 con 0x7f24140b2ee0 2024-01-16T13:53:50.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.641+0000 7f24137fe700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f24140b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:50.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.642+0000 7f2413fff700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2ee0 0x7f24140724c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f240c009d90 tx=0x7f240c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b1c92ab3550e802 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:50.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.642+0000 7f24127fc700 1 -- 172.21.15.113:0/398865853 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c00e070 con 0x7f24140b2ee0 2024-01-16T13:53:50.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.642+0000 7f24127fc700 1 -- 172.21.15.113:0/398865853 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c00bb20 con 0x7f24140b2ee0 2024-01-16T13:53:50.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.642+0000 7f24127fc700 1 -- 172.21.15.113:0/398865853 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c012430 con 0x7f24140b2ee0 2024-01-16T13:53:50.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.644+0000 7f2419d8d700 1 -- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2ee0 msgr2=0x7f24140724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:50.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.644+0000 7f2419d8d700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2ee0 0x7f24140724c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f240c009d90 tx=0x7f240c00b330 comp rx=0 tx=0).stop 2024-01-16T13:53:50.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.645+0000 7f2419d8d700 1 -- 172.21.15.113:0/398865853 shutdown_connections 2024-01-16T13:53:50.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.645+0000 7f2419d8d700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2ee0 0x7f24140724c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:50.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.645+0000 7f2419d8d700 1 --2- 172.21.15.113:0/398865853 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f24140b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:50.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.645+0000 7f2419d8d700 1 -- 172.21.15.113:0/398865853 >> 172.21.15.113:0/398865853 conn(0x7f24140b1760 msgr2=0x7f24140b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:50.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.646+0000 7f2419d8d700 1 -- 172.21.15.113:0/398865853 shutdown_connections 2024-01-16T13:53:50.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.654+0000 7f2419d8d700 1 -- 172.21.15.113:0/398865853 wait complete. 2024-01-16T13:53:50.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.656+0000 7f2419d8d700 1 Processor -- start 2024-01-16T13:53:50.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.656+0000 7f2419d8d700 1 -- start start 2024-01-16T13:53:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.657+0000 7f2419d8d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2a60 0x7f2414151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.657+0000 7f2419d8d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140b2ee0 0x7f2414150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.657+0000 7f2419d8d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f241413e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.657+0000 7f2419d8d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241413ec10 con 0x7f24140b2ee0 2024-01-16T13:53:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.657+0000 7f2419d8d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241413eef0 con 0x7f24140b2a60 2024-01-16T13:53:50.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.657+0000 7f2419d8d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241413f1d0 con 0x7f241415b320 2024-01-16T13:53:50.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.658+0000 7f2413fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f241413e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:50.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.658+0000 7f2413fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f241413e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:58004/0 (socket says 172.21.15.113:58004) 2024-01-16T13:53:50.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.658+0000 7f2413fff700 1 -- 172.21.15.113:0/3415862978 learned_addr learned my addr 172.21.15.113:0/3415862978 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.658+0000 7f2412ffd700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140b2ee0 0x7f2414150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.659+0000 7f2412ffd700 1 -- 172.21.15.113:0/3415862978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 msgr2=0x7f241413e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.659+0000 7f2412ffd700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f241413e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.659+0000 7f2412ffd700 1 -- 172.21.15.113:0/3415862978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2a60 msgr2=0x7f2414151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.659+0000 7f2412ffd700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2a60 0x7f2414151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.659+0000 7f2412ffd700 1 -- 172.21.15.113:0/3415862978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24080094a0 con 0x7f24140b2ee0 2024-01-16T13:53:50.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.659+0000 7f2412ffd700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140b2ee0 0x7f2414150c00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2404001eb0 tx=0x7f240400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:50.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.670+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404005d70 con 0x7f24140b2ee0 2024-01-16T13:53:50.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.670+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24040070d0 con 0x7f24140b2ee0 2024-01-16T13:53:50.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.670+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c009a70 con 0x7f24140b2ee0 2024-01-16T13:53:50.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.670+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404007440 con 0x7f24140b2ee0 2024-01-16T13:53:50.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.671+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24141331d0 con 0x7f24140b2ee0 2024-01-16T13:53:50.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.675+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f24040075a0 con 0x7f24140b2ee0 2024-01-16T13:53:50.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.676+0000 7f2403fff700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f23f806b550 0x7f23f806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.676+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24141333f0 con 0x7f24140b2ee0 2024-01-16T13:53:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.677+0000 7f24137fe700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f23f806b550 0x7f23f806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.677+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2404089210 con 0x7f24140b2ee0 2024-01-16T13:53:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.677+0000 7f24137fe700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f23f806b550 0x7f23f806da10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2408006f10 tx=0x7f2408006e60 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:50.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:50.683+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2404053ef0 con 0x7f24140b2ee0 2024-01-16T13:53:51.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.049+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f24141404c0 con 0x7f23f806b550 2024-01-16T13:53:51.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.072+0000 7f2403fff700 1 -- 172.21.15.113:0/3415862978 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f2408009830 con 0x7f23f806b550 2024-01-16T13:53:51.053 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "in-progress", 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "source": { 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "volume": "cephfs", 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "subvolume": "sv1", 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "snapshot": "ss1" 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "progress_report": { 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "percentage completed": 66.66666666666666, 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T13:53:51.054 INFO:teuthology.orchestra.run.smithi113.stdout: "regfiles copied": "2/3", 2024-01-16T13:53:51.055 INFO:teuthology.orchestra.run.smithi113.stdout: "subdirs copied": "0/1", 2024-01-16T13:53:51.055 INFO:teuthology.orchestra.run.smithi113.stdout: "symlinks copied": "0/0" 2024-01-16T13:53:51.055 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:51.055 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:51.055 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:51.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f23f806b550 msgr2=0x7f23f806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:51.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f23f806b550 0x7f23f806da10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2408006f10 tx=0x7f2408006e60 comp rx=0 tx=0).stop 2024-01-16T13:53:51.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140b2ee0 msgr2=0x7f2414150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:51.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140b2ee0 0x7f2414150c00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2404001eb0 tx=0x7f240400c790 comp rx=0 tx=0).stop 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 shutdown_connections 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f23f806b550 0x7f23f806da10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f241415b320 0x7f241413e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24140b2a60 0x7f2414151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 --2- 172.21.15.113:0/3415862978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140b2ee0 0x7f2414150c00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.075+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 >> 172.21.15.113:0/3415862978 conn(0x7f24140b1760 msgr2=0x7f2414072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.076+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 shutdown_connections 2024-01-16T13:53:51.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:51.076+0000 7f2419d8d700 1 -- 172.21.15.113:0/3415862978 wait complete. 2024-01-16T13:53:51.086 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T13:53:52.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T13:53:52.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.282+0000 7f834e676700 1 Processor -- start 2024-01-16T13:53:52.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.282+0000 7f834e676700 1 -- start start 2024-01-16T13:53:52.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f834e676700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f834815cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:52.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f834e676700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8348160840 0x7f8348073f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f834e676700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834815fb10 con 0x7f83480cfef0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f834e676700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8348160420 con 0x7f8348160840 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f834e676700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834815d4e0 con 0x7f834815cbc0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f8347fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f834815cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f83477fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8348160840 0x7f8348073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f8347fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f834815cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.113:58014/0 (socket says 172.21.15.113:58014) 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.283+0000 7f8347fff700 1 -- 172.21.15.113:0/226947644 learned_addr learned my addr 172.21.15.113:0/226947644 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8347fff700 1 -- 172.21.15.113:0/226947644 >> v2:172.21.15.124:3300/0 conn(0x7f8348160840 msgr2=0x7f8348073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8347fff700 1 --2- 172.21.15.113:0/226947644 >> v2:172.21.15.124:3300/0 conn(0x7f8348160840 0x7f8348073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8347fff700 1 -- 172.21.15.113:0/226947644 >> v1:172.21.15.124:6789/0 conn(0x7f83480cfef0 legacy=0x7f83480cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8347fff700 1 -- 172.21.15.113:0/226947644 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348074930 con 0x7f834815cbc0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8346ffd700 1 -- 172.21.15.113:0/226947644 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773529728 0 0) 0x7f834815fb10 con 0x7f83480cfef0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f83477fe700 1 --2- 172.21.15.113:0/226947644 >> v2:172.21.15.124:3300/0 conn(0x7f8348160840 0x7f8348073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8347fff700 1 --2- 172.21.15.113:0/226947644 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f834815cfa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f83300098f0 tx=0x7f8330009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f90d687a5a5ab483 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:52.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8346ffd700 1 -- 172.21.15.113:0/226947644 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330011830 con 0x7f834815cbc0 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8346ffd700 1 -- 172.21.15.113:0/226947644 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330011990 con 0x7f834815cbc0 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f8346ffd700 1 -- 172.21.15.113:0/226947644 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330011d20 con 0x7f834815cbc0 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 -- 172.21.15.113:0/226947644 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 msgr2=0x7f834815cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 --2- 172.21.15.113:0/226947644 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f834815cfa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f83300098f0 tx=0x7f8330009c00 comp rx=0 tx=0).stop 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 -- 172.21.15.113:0/226947644 shutdown_connections 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 --2- 172.21.15.113:0/226947644 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f834815cfa0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 --2- 172.21.15.113:0/226947644 >> v2:172.21.15.124:3300/0 conn(0x7f8348160840 0x7f8348073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 -- 172.21.15.113:0/226947644 >> 172.21.15.113:0/226947644 conn(0x7f83480cd620 msgr2=0x7f83480cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.284+0000 7f834e676700 1 -- 172.21.15.113:0/226947644 shutdown_connections 2024-01-16T13:53:52.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.285+0000 7f834e676700 1 -- 172.21.15.113:0/226947644 wait complete. 2024-01-16T13:53:52.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.286+0000 7f834e676700 1 Processor -- start 2024-01-16T13:53:52.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.286+0000 7f834e676700 1 -- start start 2024-01-16T13:53:52.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834e676700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 0x7f8348141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:52.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834e676700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f83481f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:52.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834e676700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8348160840 0x7f834813e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834e676700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834813f2b0 con 0x7f83480cfef0 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834e676700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8348141880 con 0x7f8348160840 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834e676700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8348141b60 con 0x7f834815cbc0 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f8347fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 0x7f8348141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f8347fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 0x7f8348141260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36828/0 (socket says 172.21.15.113:36828) 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f8347fff700 1 -- 172.21.15.113:0/2956626381 learned_addr learned my addr 172.21.15.113:0/2956626381 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f83477fe700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f83481f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.287+0000 7f834cc13700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8348160840 0x7f834813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8347fff700 1 -- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 msgr2=0x7f83481f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8347fff700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f83481f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8347fff700 1 -- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8348160840 msgr2=0x7f834813e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8347fff700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8348160840 0x7f834813e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:52.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8347fff700 1 -- 172.21.15.113:0/2956626381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380094a0 con 0x7f83480cfef0 2024-01-16T13:53:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8347fff700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 0x7f8348141260 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8330009c40 tx=0x7f8330004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330004510 con 0x7f83480cfef0 2024-01-16T13:53:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833001e070 con 0x7f83480cfef0 2024-01-16T13:53:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8330009550 con 0x7f83480cfef0 2024-01-16T13:53:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300038c0 con 0x7f83480cfef0 2024-01-16T13:53:52.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.288+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8348142120 con 0x7f83480cfef0 2024-01-16T13:53:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.289+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834815ebb0 con 0x7f83480cfef0 2024-01-16T13:53:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.289+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f8330003a20 con 0x7f83480cfef0 2024-01-16T13:53:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.290+0000 7f8344ff9700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f832806b500 0x7f832806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.290+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8330020070 con 0x7f83480cfef0 2024-01-16T13:53:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.290+0000 7f83477fe700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f832806b500 0x7f832806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:52.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.290+0000 7f83477fe700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f832806b500 0x7f832806d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f83481f6100 tx=0x7f8338009210 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:52.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.294+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83300bde90 con 0x7f83480cfef0 2024-01-16T13:53:52.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:52.531+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8348160040 con 0x7f832806b500 2024-01-16T13:53:53.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.284+0000 7f8344ff9700 1 -- 172.21.15.113:0/2956626381 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f8338009920 con 0x7f832806b500 2024-01-16T13:53:53.266 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-01-16T13:53:53.267 INFO:teuthology.orchestra.run.smithi113.stdout: "status": { 2024-01-16T13:53:53.267 INFO:teuthology.orchestra.run.smithi113.stdout: "state": "complete" 2024-01-16T13:53:53.267 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:53.267 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f832806b500 msgr2=0x7f832806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f832806b500 0x7f832806d9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f83481f6100 tx=0x7f8338009210 comp rx=0 tx=0).stop 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 msgr2=0x7f8348141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 0x7f8348141260 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8330009c40 tx=0x7f8330004750 comp rx=0 tx=0).stop 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 shutdown_connections 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f832806b500 0x7f832806d9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8348160840 0x7f834813e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83480cfef0 0x7f8348141260 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 --2- 172.21.15.113:0/2956626381 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f834815cbc0 0x7f83481f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 >> 172.21.15.113:0/2956626381 conn(0x7f83480cd620 msgr2=0x7f8348071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 shutdown_connections 2024-01-16T13:53:53.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.287+0000 7f834e676700 1 -- 172.21.15.113:0/2956626381 wait complete. 2024-01-16T13:53:53.282 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T13:53:53.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.424+0000 7f3ed3d8d700 1 Processor -- start 2024-01-16T13:53:53.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.424+0000 7f3ed3d8d700 1 -- start start 2024-01-16T13:53:53.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed3d8d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc1446a0 0x7f3ecc0b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed3d8d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3ecc0adf00 0x7f3ecc06c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed3d8d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ecc0af950 con 0x7f3ecc06c540 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed3d8d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ecc144a80 con 0x7f3ecc0adf00 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed3d8d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc06c9b0 con 0x7f3ecc1446a0 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed1b29700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc1446a0 0x7f3ecc0b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed1328700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3ecc0adf00 0x7f3ecc06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed232a700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f3ecc06c540 0x7f3ecc06da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:35290/0 (socket says 172.21.15.113:35290) 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed232a700 1 -- 172.21.15.113:0/1662798850 learned_addr learned my addr 172.21.15.113:0/1662798850 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed1328700 1 -- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc1446a0 msgr2=0x7f3ecc0b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed1328700 1 --2- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc1446a0 0x7f3ecc0b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed1328700 1 -- 172.21.15.113:0/1662798850 >> v1:172.21.15.124:6789/0 conn(0x7f3ecc06c540 legacy=0x7f3ecc06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed1328700 1 -- 172.21.15.113:0/1662798850 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc0afc30 con 0x7f3ecc0adf00 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.425+0000 7f3ed0b27700 1 -- 172.21.15.113:0/1662798850 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622055970 0 0) 0x7f3ecc0af950 con 0x7f3ecc06c540 2024-01-16T13:53:53.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed1328700 1 --2- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc0adf00 0x7f3ecc06c000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009870 tx=0x7f3ec8009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb78b7c6d71c7e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:53.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed0b27700 1 -- 172.21.15.113:0/1662798850 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011830 con 0x7f3ecc0adf00 2024-01-16T13:53:53.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed0b27700 1 -- 172.21.15.113:0/1662798850 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8011990 con 0x7f3ecc0adf00 2024-01-16T13:53:53.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed0b27700 1 -- 172.21.15.113:0/1662798850 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011d20 con 0x7f3ecc0adf00 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc0adf00 msgr2=0x7f3ecc06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc0adf00 0x7f3ecc06c000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009870 tx=0x7f3ec8009b80 comp rx=0 tx=0).stop 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/1662798850 shutdown_connections 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc1446a0 0x7f3ecc0b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/1662798850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc0adf00 0x7f3ecc06c000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/1662798850 >> 172.21.15.113:0/1662798850 conn(0x7f3ecc071b90 msgr2=0x7f3ecc071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/1662798850 shutdown_connections 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.426+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/1662798850 wait complete. 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 Processor -- start 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 -- start start 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc06c540 0x7f3ecc1566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc0adf00 0x7f3ecc142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 0x7f3ecc0b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc1f4150 con 0x7f3ecc1446a0 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc1f4430 con 0x7f3ecc06c540 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed3d8d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc1f4710 con 0x7f3ecc0adf00 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed232a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 0x7f3ecc0b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.428+0000 7f3ed232a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 0x7f3ecc0b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47866/0 (socket says 172.21.15.113:47866) 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 -- 172.21.15.113:0/2670884482 learned_addr learned my addr 172.21.15.113:0/2670884482 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed1328700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc0adf00 0x7f3ecc142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed1b29700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc06c540 0x7f3ecc1566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 -- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc0adf00 msgr2=0x7f3ecc142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc0adf00 0x7f3ecc142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 -- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc06c540 msgr2=0x7f3ecc1566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc06c540 0x7f3ecc1566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 -- 172.21.15.113:0/2670884482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec80094a0 con 0x7f3ecc1446a0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed1328700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc0adf00 0x7f3ecc142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed232a700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 0x7f3ecc0b3bc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4002230 tx=0x7f3ec400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400ac20 con 0x7f3ecc1446a0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec400ada0 con 0x7f3ecc1446a0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ecc1f49f0 con 0x7f3ecc1446a0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4007310 con 0x7f3ecc1446a0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.429+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ecc1425a0 con 0x7f3ecc1446a0 2024-01-16T13:53:53.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.430+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ecc142970 con 0x7f3ecc1446a0 2024-01-16T13:53:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.431+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f3ec4007470 con 0x7f3ecc1446a0 2024-01-16T13:53:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.431+0000 7f3ebe7fc700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3eb006b530 0x7f3eb006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.431+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec4089990 con 0x7f3ecc1446a0 2024-01-16T13:53:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.431+0000 7f3ed1b29700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3eb006b530 0x7f3eb006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.431+0000 7f3ed1b29700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3eb006b530 0x7f3eb006d9f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009ad0 tx=0x7f3ec0008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:53.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.435+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ec4054590 con 0x7f3ecc1446a0 2024-01-16T13:53:53.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.654+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ecc0b04d0 con 0x7f3eb006b530 2024-01-16T13:53:53.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.655+0000 7f3ebe7fc700 1 -- 172.21.15.113:0/2670884482 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3ec0009720 con 0x7f3eb006b530 2024-01-16T13:53:53.637 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-16T13:53:53.637 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:53.637 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "cephfs" 2024-01-16T13:53:53.637 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:53.637 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-16T13:53:53.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3eb006b530 msgr2=0x7f3eb006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3eb006b530 0x7f3eb006d9f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009ad0 tx=0x7f3ec0008040 comp rx=0 tx=0).stop 2024-01-16T13:53:53.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 msgr2=0x7f3ecc0b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 0x7f3ecc0b3bc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4002230 tx=0x7f3ec400a060 comp rx=0 tx=0).stop 2024-01-16T13:53:53.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 shutdown_connections 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f3eb006b530 0x7f3eb006d9f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ecc1446a0 0x7f3ecc0b3bc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ecc06c540 0x7f3ecc1566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 --2- 172.21.15.113:0/2670884482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3ecc0adf00 0x7f3ecc142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 >> 172.21.15.113:0/2670884482 conn(0x7f3ecc071b90 msgr2=0x7f3ecc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 shutdown_connections 2024-01-16T13:53:53.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.657+0000 7f3ed3d8d700 1 -- 172.21.15.113:0/2670884482 wait complete. 2024-01-16T13:53:53.651 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T13:53:53.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.792+0000 7f54b2fac700 1 Processor -- start 2024-01-16T13:53:53.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.792+0000 7f54b2fac700 1 -- start start 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54b2fac700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54ac06f680 0x7f54ac0afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54b2fac700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac06fb30 con 0x7f54ac15e110 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54b2fac700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac06d980 con 0x7f54ac06e420 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54b2fac700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54ac0b0070 con 0x7f54ac06f680 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54b0d48700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f54ac15e110 0x7f54ac06aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40724/0 (socket says 172.21.15.113:40724) 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54b0d48700 1 -- 172.21.15.113:0/2635357297 learned_addr learned my addr 172.21.15.113:0/2635357297 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54abfff700 1 --2- 172.21.15.113:0/2635357297 >> v2:172.21.15.124:3300/0 conn(0x7f54ac06f680 0x7f54ac0afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229451980 0 0) 0x7f54ac06fb30 con 0x7f54ac15e110 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549c0035b0 con 0x7f54ac15e110 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571353405 0 0) 0x7f54ac06d980 con 0x7f54ac06e420 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.793+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac06fb30 con 0x7f54ac06e420 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2095877820 0 0) 0x7f549c0035b0 con 0x7f54ac15e110 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54abfff700 1 -- 172.21.15.113:0/2635357297 >> v1:172.21.15.124:6789/0 conn(0x7f54ac06e420 legacy=0x7f54ac073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54abfff700 1 -- 172.21.15.113:0/2635357297 >> v1:172.21.15.113:6789/0 conn(0x7f54ac15e110 legacy=0x7f54ac06aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54abfff700 1 -- 172.21.15.113:0/2635357297 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac0b0260 con 0x7f54ac06f680 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54abfff700 1 --2- 172.21.15.113:0/2635357297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0afb30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5498009840 tx=0x7f5498009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90543900d1865ce1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0001670 con 0x7f54ac06f680 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498004030 con 0x7f54ac06f680 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54ab7fe700 1 -- 172.21.15.113:0/2635357297 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54980043c0 con 0x7f54ac06f680 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54b2fac700 1 -- 172.21.15.113:0/2635357297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 msgr2=0x7f54ac0afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54b2fac700 1 --2- 172.21.15.113:0/2635357297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0afb30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5498009840 tx=0x7f5498009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54b2fac700 1 -- 172.21.15.113:0/2635357297 shutdown_connections 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54b2fac700 1 --2- 172.21.15.113:0/2635357297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0afb30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.794+0000 7f54b2fac700 1 -- 172.21.15.113:0/2635357297 >> 172.21.15.113:0/2635357297 conn(0x7f54ac0b3250 msgr2=0x7f54ac0d0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.795+0000 7f54b2fac700 1 -- 172.21.15.113:0/2635357297 shutdown_connections 2024-01-16T13:53:53.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.795+0000 7f54b2fac700 1 -- 172.21.15.113:0/2635357297 wait complete. 2024-01-16T13:53:53.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.796+0000 7f54b2fac700 1 Processor -- start 2024-01-16T13:53:53.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.796+0000 7f54b2fac700 1 -- start start 2024-01-16T13:53:53.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.796+0000 7f54b2fac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54ac06e420 0x7f54ac14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 0x7f54ac0c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0c71f0 con 0x7f54ac15e110 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0c65e0 con 0x7f54ac06f680 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0c68c0 con 0x7f54ac06e420 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 0x7f54ac0c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54abfff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54abfff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.113:36288/0 (socket says 172.21.15.113:36288) 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 0x7f54ac0c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47878/0 (socket says 172.21.15.113:47878) 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54abfff700 1 -- 172.21.15.113:0/4171838436 learned_addr learned my addr 172.21.15.113:0/4171838436 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b0d48700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54ac06e420 0x7f54ac14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 -- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54ac06e420 msgr2=0x7f54ac14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54ac06e420 0x7f54ac14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 -- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 msgr2=0x7f54ac0c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:53.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 -- 172.21.15.113:0/4171838436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980094a0 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b1549700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 0x7f54ac0c5b60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f54a400cb10 tx=0x7f54a400ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4003c00 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4003d60 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac0c6ba0 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a400aa20 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.797+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac14ea80 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.798+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac069020 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.799+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f54a400ab80 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.799+0000 7f54a97fa700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f549006b500 0x7f549006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.799+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54a4089e00 con 0x7f54ac15e110 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.799+0000 7f54b0d48700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f549006b500 0x7f549006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:53.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.799+0000 7f54b0d48700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f549006b500 0x7f549006d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f54a00011e0 tx=0x7f54a0000ed0 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:53.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:53.804+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54a4054960 con 0x7f54ac15e110 2024-01-16T13:53:54.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.023+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f54ac06f0e0 con 0x7f549006b500 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.027+0000 7f54a97fa700 1 -- 172.21.15.113:0/4171838436 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f54ac06f0e0 con 0x7f549006b500 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "sv1" 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-01-16T13:53:54.009 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:54.010 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "ss1clone1" 2024-01-16T13:53:54.010 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:54.010 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-16T13:53:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f549006b500 msgr2=0x7f549006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f549006b500 0x7f549006d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f54a00011e0 tx=0x7f54a0000ed0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 msgr2=0x7f54ac0c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 0x7f54ac0c5b60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f54a400cb10 tx=0x7f54a400ce20 comp rx=0 tx=0).stop 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 shutdown_connections 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f549006b500 0x7f549006d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac15e110 0x7f54ac0c5b60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54ac06e420 0x7f54ac14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 --2- 172.21.15.113:0/4171838436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54ac06f680 0x7f54ac0c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 >> 172.21.15.113:0/4171838436 conn(0x7f54ac0b3250 msgr2=0x7f54ac15d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 shutdown_connections 2024-01-16T13:53:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.029+0000 7f54b2fac700 1 -- 172.21.15.113:0/4171838436 wait complete. 2024-01-16T13:53:54.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T13:53:54.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.167+0000 7f6a3dea4700 1 Processor -- start 2024-01-16T13:53:54.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.167+0000 7f6a3dea4700 1 -- start start 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a3dea4700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6a380cf040 0x7f6a3806fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a3dea4700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a380af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a3dea4700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380d01c0 con 0x7f6a380702e0 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a3dea4700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a38070910 con 0x7f6a380cf040 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a3dea4700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38070bf0 con 0x7f6a380d21c0 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a37fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a380af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a377fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6a380cf040 0x7f6a3806fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a36ffd700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6a380702e0 0x7f6a380d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:35312/0 (socket says 172.21.15.113:35312) 2024-01-16T13:53:54.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.168+0000 7f6a36ffd700 1 -- 172.21.15.113:0/1684537247 learned_addr learned my addr 172.21.15.113:0/1684537247 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a367fc700 1 -- 172.21.15.113:0/1684537247 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259917443 0 0) 0x7f6a380d01c0 con 0x7f6a380702e0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a367fc700 1 -- 172.21.15.113:0/1684537247 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a200035b0 con 0x7f6a380702e0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a37fff700 1 -- 172.21.15.113:0/1684537247 >> v2:172.21.15.124:3300/0 conn(0x7f6a380cf040 msgr2=0x7f6a3806fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a37fff700 1 --2- 172.21.15.113:0/1684537247 >> v2:172.21.15.124:3300/0 conn(0x7f6a380cf040 0x7f6a3806fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a37fff700 1 -- 172.21.15.113:0/1684537247 >> v1:172.21.15.124:6789/0 conn(0x7f6a380702e0 legacy=0x7f6a380d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a37fff700 1 -- 172.21.15.113:0/1684537247 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380aebb0 con 0x7f6a380d21c0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a377fe700 1 --2- 172.21.15.113:0/1684537247 >> v2:172.21.15.124:3300/0 conn(0x7f6a380cf040 0x7f6a3806fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a37fff700 1 --2- 172.21.15.113:0/1684537247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a380af3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009d90 tx=0x7f6a3000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd15365b82f55817 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a367fc700 1 -- 172.21.15.113:0/1684537247 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000e070 con 0x7f6a380d21c0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a367fc700 1 -- 172.21.15.113:0/1684537247 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3000bb20 con 0x7f6a380d21c0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.169+0000 7f6a367fc700 1 -- 172.21.15.113:0/1684537247 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000be90 con 0x7f6a380d21c0 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 -- 172.21.15.113:0/1684537247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 msgr2=0x7f6a380af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/1684537247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a380af3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009d90 tx=0x7f6a3000b330 comp rx=0 tx=0).stop 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 -- 172.21.15.113:0/1684537247 shutdown_connections 2024-01-16T13:53:54.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/1684537247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a380af3b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/1684537247 >> v2:172.21.15.124:3300/0 conn(0x7f6a380cf040 0x7f6a3806fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 -- 172.21.15.113:0/1684537247 >> 172.21.15.113:0/1684537247 conn(0x7f6a3809aa00 msgr2=0x7f6a3809ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:54.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 -- 172.21.15.113:0/1684537247 shutdown_connections 2024-01-16T13:53:54.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.170+0000 7f6a3dea4700 1 -- 172.21.15.113:0/1684537247 wait complete. 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.171+0000 7f6a3dea4700 1 Processor -- start 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.171+0000 7f6a3dea4700 1 -- start start 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a3dea4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 0x7f6a3815b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a3dea4700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380cf040 0x7f6a380b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a3dea4700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a381f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a3dea4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3815b750 con 0x7f6a380702e0 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a3dea4700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380b47f0 con 0x7f6a380cf040 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a3dea4700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380cbc30 con 0x7f6a380d21c0 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 0x7f6a3815b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 0x7f6a3815b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47882/0 (socket says 172.21.15.113:47882) 2024-01-16T13:53:54.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 -- 172.21.15.113:0/2025827890 learned_addr learned my addr 172.21.15.113:0/2025827890 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a36ffd700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380cf040 0x7f6a380b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a37fff700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a381f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 -- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 msgr2=0x7f6a381f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a381f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 -- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380cf040 msgr2=0x7f6a380b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380cf040 0x7f6a380b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 -- 172.21.15.113:0/2025827890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0094a0 con 0x7f6a380702e0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.172+0000 7f6a377fe700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 0x7f6a3815b210 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00ebe0 tx=0x7f6a2c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.173+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00c850 con 0x7f6a380702e0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.173+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c00c9b0 con 0x7f6a380702e0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.173+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a30009a70 con 0x7f6a380702e0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.173+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00cd40 con 0x7f6a380702e0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.173+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a380b4d20 con 0x7f6a380702e0 2024-01-16T13:53:54.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.174+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a38067f60 con 0x7f6a380702e0 2024-01-16T13:53:54.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.174+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f6a2c013470 con 0x7f6a380702e0 2024-01-16T13:53:54.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.175+0000 7f6a1ffff700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b500 0x7f6a2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.175+0000 7f6a36ffd700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b500 0x7f6a2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.175+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a2c08a2c0 con 0x7f6a380702e0 2024-01-16T13:53:54.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.175+0000 7f6a36ffd700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b500 0x7f6a2406d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a380b2e20 tx=0x7f6a28009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.179+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a2c054eb0 con 0x7f6a380702e0 2024-01-16T13:53:54.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.404+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a3806dfe0 con 0x7f6a2406b500 2024-01-16T13:53:54.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.416+0000 7f6a1ffff700 1 -- 172.21.15.113:0/2025827890 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f6a3806dfe0 con 0x7f6a2406b500 2024-01-16T13:53:54.398 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:54.398 INFO:teuthology.orchestra.run.smithi113.stdout:[ 2024-01-16T13:53:54.398 INFO:teuthology.orchestra.run.smithi113.stdout: { 2024-01-16T13:53:54.398 INFO:teuthology.orchestra.run.smithi113.stdout: "name": "ss1" 2024-01-16T13:53:54.398 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-01-16T13:53:54.399 INFO:teuthology.orchestra.run.smithi113.stdout:] 2024-01-16T13:53:54.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b500 msgr2=0x7f6a2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b500 0x7f6a2406d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a380b2e20 tx=0x7f6a28009040 comp rx=0 tx=0).stop 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 msgr2=0x7f6a3815b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 0x7f6a3815b210 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00ebe0 tx=0x7f6a2c00eef0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 shutdown_connections 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6a2406b500 0x7f6a2406d9c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6a380d21c0 0x7f6a381f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a380702e0 0x7f6a3815b210 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 --2- 172.21.15.113:0/2025827890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a380cf040 0x7f6a380b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 >> 172.21.15.113:0/2025827890 conn(0x7f6a3809aa00 msgr2=0x7f6a380d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 shutdown_connections 2024-01-16T13:53:54.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.418+0000 7f6a3dea4700 1 -- 172.21.15.113:0/2025827890 wait complete. 2024-01-16T13:53:54.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T13:53:54.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.559+0000 7f6e64470700 1 Processor -- start 2024-01-16T13:53:54.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.559+0000 7f6e64470700 1 -- start start 2024-01-16T13:53:54.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e64470700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6e5c15bee0 0x7f6e5c0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e64470700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0d0790 con 0x7f6e5c0cf020 2024-01-16T13:53:54.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e64470700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c06ee70 con 0x7f6e5c0ceb20 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e64470700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5c06f150 con 0x7f6e5c15bee0 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e61a0b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6e5c0cf020 0x7f6e5c0ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40730/0 (socket says 172.21.15.113:40730) 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e61a0b700 1 -- 172.21.15.113:0/4047107832 learned_addr learned my addr 172.21.15.113:0/4047107832 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6220c700 1 --2- 172.21.15.113:0/4047107832 >> v2:172.21.15.124:3300/0 conn(0x7f6e5c15bee0 0x7f6e5c0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6120a700 1 -- 172.21.15.113:0/4047107832 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407951198 0 0) 0x7f6e5c0d0790 con 0x7f6e5c0cf020 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6120a700 1 -- 172.21.15.113:0/4047107832 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e480035b0 con 0x7f6e5c0cf020 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6220c700 1 -- 172.21.15.113:0/4047107832 >> v1:172.21.15.124:6789/0 conn(0x7f6e5c0ceb20 legacy=0x7f6e5c06f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6120a700 1 -- 172.21.15.113:0/4047107832 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514812524 0 0) 0x7f6e5c06ee70 con 0x7f6e5c0ceb20 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6220c700 1 -- 172.21.15.113:0/4047107832 >> v1:172.21.15.113:6789/0 conn(0x7f6e5c0cf020 legacy=0x7f6e5c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.560+0000 7f6e6220c700 1 -- 172.21.15.113:0/4047107832 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c06f430 con 0x7f6e5c15bee0 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.561+0000 7f6e6220c700 1 --2- 172.21.15.113:0/4047107832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c15bee0 0x7f6e5c0d01b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009840 tx=0x7f6e58009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3567fee5277f9746 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.561+0000 7f6e6120a700 1 -- 172.21.15.113:0/4047107832 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58011830 con 0x7f6e5c15bee0 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.561+0000 7f6e6120a700 1 -- 172.21.15.113:0/4047107832 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e58011990 con 0x7f6e5c15bee0 2024-01-16T13:53:54.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.561+0000 7f6e6120a700 1 -- 172.21.15.113:0/4047107832 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58011d20 con 0x7f6e5c15bee0 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.561+0000 7f6e64470700 1 -- 172.21.15.113:0/4047107832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c15bee0 msgr2=0x7f6e5c0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.561+0000 7f6e64470700 1 --2- 172.21.15.113:0/4047107832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c15bee0 0x7f6e5c0d01b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009840 tx=0x7f6e58009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.562+0000 7f6e64470700 1 -- 172.21.15.113:0/4047107832 shutdown_connections 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.562+0000 7f6e64470700 1 --2- 172.21.15.113:0/4047107832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c15bee0 0x7f6e5c0d01b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.562+0000 7f6e64470700 1 -- 172.21.15.113:0/4047107832 >> 172.21.15.113:0/4047107832 conn(0x7f6e5c0b3780 msgr2=0x7f6e5c0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.562+0000 7f6e64470700 1 -- 172.21.15.113:0/4047107832 shutdown_connections 2024-01-16T13:53:54.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.562+0000 7f6e64470700 1 -- 172.21.15.113:0/4047107832 wait complete. 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.563+0000 7f6e64470700 1 Processor -- start 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.563+0000 7f6e64470700 1 -- start start 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e64470700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c0ceb20 0x7f6e5c1f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e64470700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 0x7f6e5c154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e64470700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6e5c15bee0 0x7f6e5c15e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e64470700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1f2870 con 0x7f6e5c0cf020 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e64470700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1f51d0 con 0x7f6e5c0ceb20 2024-01-16T13:53:54.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e64470700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1f54b0 con 0x7f6e5c15bee0 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 0x7f6e5c154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 0x7f6e5c154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47888/0 (socket says 172.21.15.113:47888) 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 -- 172.21.15.113:0/2291719719 learned_addr learned my addr 172.21.15.113:0/2291719719 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e6220c700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c0ceb20 0x7f6e5c1f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e62a0d700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6e5c15bee0 0x7f6e5c15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 -- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6e5c15bee0 msgr2=0x7f6e5c15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6e5c15bee0 0x7f6e5c15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 -- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c0ceb20 msgr2=0x7f6e5c1f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c0ceb20 0x7f6e5c1f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.564+0000 7f6e61a0b700 1 -- 172.21.15.113:0/2291719719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e580094a0 con 0x7f6e5c0cf020 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.565+0000 7f6e61a0b700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 0x7f6e5c154100 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00c4d0 tx=0x7f6e4c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.565+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c005d70 con 0x7f6e5c0cf020 2024-01-16T13:53:54.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.565+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c0070d0 con 0x7f6e5c0cf020 2024-01-16T13:53:54.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.565+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c1f5790 con 0x7f6e5c0cf020 2024-01-16T13:53:54.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.565+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c007440 con 0x7f6e5c0cf020 2024-01-16T13:53:54.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.565+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c1f5d50 con 0x7f6e5c0cf020 2024-01-16T13:53:54.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.566+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c0c5630 con 0x7f6e5c0cf020 2024-01-16T13:53:54.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.566+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f6e4c0075a0 con 0x7f6e5c0cf020 2024-01-16T13:53:54.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.567+0000 7f6e52ffd700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6e4006b500 0x7f6e4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.567+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e4c089170 con 0x7f6e5c0cf020 2024-01-16T13:53:54.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.567+0000 7f6e6220c700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6e4006b500 0x7f6e4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.567+0000 7f6e6220c700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6e4006b500 0x7f6e4006d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009fd0 tx=0x7f6e5801c040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.571+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e4c053e10 con 0x7f6e5c0cf020 2024-01-16T13:53:54.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.806+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e5c0c5440 con 0x7f6e4006b500 2024-01-16T13:53:54.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.833+0000 7f6e52ffd700 1 -- 172.21.15.113:0/2291719719 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6e5c0c5440 con 0x7f6e4006b500 2024-01-16T13:53:54.815 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:54.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6e4006b500 msgr2=0x7f6e4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6e4006b500 0x7f6e4006d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009fd0 tx=0x7f6e5801c040 comp rx=0 tx=0).stop 2024-01-16T13:53:54.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 msgr2=0x7f6e5c154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 0x7f6e5c154100 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00c4d0 tx=0x7f6e4c00c7e0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 shutdown_connections 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f6e4006b500 0x7f6e4006d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6e5c15bee0 0x7f6e5c15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e5c0ceb20 0x7f6e5c1f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 --2- 172.21.15.113:0/2291719719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e5c0cf020 0x7f6e5c154100 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 >> 172.21.15.113:0/2291719719 conn(0x7f6e5c0b3780 msgr2=0x7f6e5c0b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 shutdown_connections 2024-01-16T13:53:54.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.835+0000 7f6e64470700 1 -- 172.21.15.113:0/2291719719 wait complete. 2024-01-16T13:53:54.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T13:53:54.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.971+0000 7ffa94fb7700 1 Processor -- start 2024-01-16T13:53:54.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.971+0000 7ffa94fb7700 1 -- start start 2024-01-16T13:53:54.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.972+0000 7ffa94fb7700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ffa9015e280 0x7ffa900b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.972+0000 7ffa94fb7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 0x7ffa900b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.972+0000 7ffa94fb7700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa900b42a0 con 0x7ffa900b5050 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.972+0000 7ffa94fb7700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa900b5500 con 0x7ffa9015e280 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.972+0000 7ffa94fb7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa900af900 con 0x7ffa90073f90 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8ed9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ffa900b5050 0x7ffa900ae4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40742/0 (socket says 172.21.15.113:40742) 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8ed9d700 1 -- 172.21.15.113:0/1808159287 learned_addr learned my addr 172.21.15.113:0/1808159287 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8dd9b700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 0x7ffa900b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8e59c700 1 --2- 172.21.15.113:0/1808159287 >> v2:172.21.15.124:3300/0 conn(0x7ffa9015e280 0x7ffa900b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225392533 0 0) 0x7ffa88001490 con 0x7ffa900b5050 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c0035b0 con 0x7ffa900b5050 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508280565 0 0) 0x7ffa7c0035b0 con 0x7ffa900b5050 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 msgr2=0x7ffa900b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 0x7ffa900b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 >> v2:172.21.15.124:3300/0 conn(0x7ffa9015e280 msgr2=0x7ffa900b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:54.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 --2- 172.21.15.113:0/1808159287 >> v2:172.21.15.124:3300/0 conn(0x7ffa9015e280 0x7ffa900b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90074d30 con 0x7ffa900b5050 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7ffa88001b80 con 0x7ffa900b5050 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 >> v1:172.21.15.113:6789/0 conn(0x7ffa900b5050 legacy=0x7ffa900ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 0x7ffa7c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa7c003f80 0x7ffa7c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa7c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa900b5500 con 0x7ffa7c004d90 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa900af900 con 0x7ffa7c003f80 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c00ba10 con 0x7ffa90073f90 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8ed9d700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa7c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8e59c700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 0x7ffa7c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.973+0000 7ffa8dd9b700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa7c003f80 0x7ffa7c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 -- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 msgr2=0x7ffa7c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa90073f90 0x7ffa7c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 -- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa7c003f80 msgr2=0x7ffa7c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa7c003f80 0x7ffa7c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 -- 172.21.15.113:0/1808159287 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c0035b0 con 0x7ffa7c004d90 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa7c005210 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800a830 tx=0x7ffa7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a455b5dd5cd95ea3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8800a4c0 con 0x7ffa7c004d90 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88012600 con 0x7ffa7c004d90 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8d59a700 1 -- 172.21.15.113:0/1808159287 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa880129b0 con 0x7ffa7c004d90 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa94fb7700 1 -- 172.21.15.113:0/1808159287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 msgr2=0x7ffa7c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/1808159287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa7c005210 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800a830 tx=0x7ffa7c005880 comp rx=0 tx=0).stop 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa8ed9d700 1 -- 172.21.15.113:0/1808159287 reap_dead start 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa94fb7700 1 -- 172.21.15.113:0/1808159287 shutdown_connections 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.974+0000 7ffa94fb7700 1 -- 172.21.15.113:0/1808159287 >> 172.21.15.113:0/1808159287 conn(0x7ffa90072b20 msgr2=0x7ffa9006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.975+0000 7ffa94fb7700 1 -- 172.21.15.113:0/1808159287 shutdown_connections 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.975+0000 7ffa94fb7700 1 -- 172.21.15.113:0/1808159287 wait complete. 2024-01-16T13:53:54.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.976+0000 7ffa94fb7700 1 Processor -- start 2024-01-16T13:53:54.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.976+0000 7ffa94fb7700 1 -- start start 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa90156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa901530e0 0x7ffa901528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa901593b0 0x7ffa90159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90159da0 con 0x7ffa7c004d90 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa90153de0 con 0x7ffa901530e0 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa901f3aa0 con 0x7ffa901593b0 2024-01-16T13:53:54.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa90156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa90156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47900/0 (socket says 172.21.15.113:47900) 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 -- 172.21.15.113:0/3936613818 learned_addr learned my addr 172.21.15.113:0/3936613818 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8ed9d700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa901593b0 0x7ffa90159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8dd9b700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa901530e0 0x7ffa901528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 -- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa901593b0 msgr2=0x7ffa90159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa901593b0 0x7ffa90159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 -- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa901530e0 msgr2=0x7ffa901528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa901530e0 0x7ffa901528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 -- 172.21.15.113:0/3936613818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa901f4820 con 0x7ffa7c004d90 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa8e59c700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa90156060 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ffa78011aa0 tx=0x7ffa78011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7800f9e0 con 0x7ffa7c004d90 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7800fb40 con 0x7ffa7c004d90 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa901f3d40 con 0x7ffa7c004d90 2024-01-16T13:53:54.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa78014440 con 0x7ffa7c004d90 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.977+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa901f4300 con 0x7ffa7c004d90 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.978+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa90069020 con 0x7ffa7c004d90 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.979+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7ffa780145a0 con 0x7ffa7c004d90 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.979+0000 7ffa86ffd700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa7006b500 0x7ffa7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.979+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa7808a6b0 con 0x7ffa7c004d90 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.979+0000 7ffa8dd9b700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa7006b500 0x7ffa7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.980+0000 7ffa8dd9b700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa7006b500 0x7ffa7006d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffa9006fa40 tx=0x7ffa80008040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:54.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:54.984+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa780552a0 con 0x7ffa7c004d90 2024-01-16T13:53:55.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.202+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa90074a50 con 0x7ffa7006b500 2024-01-16T13:53:55.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.220+0000 7ffa86ffd700 1 -- 172.21.15.113:0/3936613818 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffa80014840 con 0x7ffa7006b500 2024-01-16T13:53:55.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa7006b500 msgr2=0x7ffa7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa7006b500 0x7ffa7006d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffa9006fa40 tx=0x7ffa80008040 comp rx=0 tx=0).stop 2024-01-16T13:53:55.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 msgr2=0x7ffa90156060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa90156060 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ffa78011aa0 tx=0x7ffa78011e60 comp rx=0 tx=0).stop 2024-01-16T13:53:55.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 shutdown_connections 2024-01-16T13:53:55.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7ffa7006b500 0x7ffa7006d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ffa901593b0 0x7ffa90159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa7c004d90 0x7ffa90156060 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 --2- 172.21.15.113:0/3936613818 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffa901530e0 0x7ffa901528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 >> 172.21.15.113:0/3936613818 conn(0x7ffa90072b20 msgr2=0x7ffa9006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 shutdown_connections 2024-01-16T13:53:55.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.222+0000 7ffa94fb7700 1 -- 172.21.15.113:0/3936613818 wait complete. 2024-01-16T13:53:55.207 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.223+0000 7fa58b565700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.207 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.223+0000 7fa58b565700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.207 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.223+0000 7fa58b565700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.207 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.223+0000 7fa58b565700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.207 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.223+0000 7fa58b565700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T13:53:55.220 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.238+0000 7fa58a563700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.220 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.238+0000 7fa58a563700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.221 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.238+0000 7fa58a563700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.221 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.238+0000 7fa58a563700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.221 INFO:tasks.ceph.mgr.y.smithi113.stderr:2024-01-16T13:53:55.238+0000 7fa58a563700 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T13:53:55.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.367+0000 7fed9091d700 1 Processor -- start 2024-01-16T13:53:55.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.367+0000 7fed9091d700 1 -- start start 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed9091d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c15e0c0 0x7fed8c0cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed9091d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fed8c071d20 0x7fed8c0b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed9091d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8c0d29e0 con 0x7fed8c0d3a40 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed9091d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c0d2c80 con 0x7fed8c071d20 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed9091d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c0d2f60 con 0x7fed8c15e0c0 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8ad9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fed8c071d20 0x7fed8c0b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c15e0c0 0x7fed8c0cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed89d9b700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fed8c0d3a40 0x7fed8c071610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.113:35326/0 (socket says 172.21.15.113:35326) 2024-01-16T13:53:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed89d9b700 1 -- 172.21.15.113:0/3322317370 learned_addr learned my addr 172.21.15.113:0/3322317370 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8a59c700 1 -- 172.21.15.113:0/3322317370 >> v2:172.21.15.124:3300/0 conn(0x7fed8c071d20 msgr2=0x7fed8c0b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8a59c700 1 --2- 172.21.15.113:0/3322317370 >> v2:172.21.15.124:3300/0 conn(0x7fed8c071d20 0x7fed8c0b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8a59c700 1 -- 172.21.15.113:0/3322317370 >> v1:172.21.15.124:6789/0 conn(0x7fed8c0d3a40 legacy=0x7fed8c071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8a59c700 1 -- 172.21.15.113:0/3322317370 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c0d3240 con 0x7fed8c15e0c0 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.368+0000 7fed8ad9d700 1 --2- 172.21.15.113:0/3322317370 >> v2:172.21.15.124:3300/0 conn(0x7fed8c071d20 0x7fed8c0b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed8a59c700 1 --2- 172.21.15.113:0/3322317370 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c15e0c0 0x7fed8c0cec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009840 tx=0x7fed7c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e879d12709412adf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed8959a700 1 -- 172.21.15.113:0/3322317370 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c011830 con 0x7fed8c15e0c0 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed8959a700 1 -- 172.21.15.113:0/3322317370 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c011990 con 0x7fed8c15e0c0 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed8959a700 1 -- 172.21.15.113:0/3322317370 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c011d20 con 0x7fed8c15e0c0 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 -- 172.21.15.113:0/3322317370 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c15e0c0 msgr2=0x7fed8c0cec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 --2- 172.21.15.113:0/3322317370 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c15e0c0 0x7fed8c0cec80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009840 tx=0x7fed7c009b50 comp rx=0 tx=0).stop 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 -- 172.21.15.113:0/3322317370 shutdown_connections 2024-01-16T13:53:55.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 --2- 172.21.15.113:0/3322317370 >> v2:172.21.15.124:3300/0 conn(0x7fed8c071d20 0x7fed8c0b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 --2- 172.21.15.113:0/3322317370 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c15e0c0 0x7fed8c0cec80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 -- 172.21.15.113:0/3322317370 >> 172.21.15.113:0/3322317370 conn(0x7fed8c06b620 msgr2=0x7fed8c0d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:55.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 -- 172.21.15.113:0/3322317370 shutdown_connections 2024-01-16T13:53:55.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.369+0000 7fed9091d700 1 -- 172.21.15.113:0/3322317370 wait complete. 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.371+0000 7fed9091d700 1 Processor -- start 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.371+0000 7fed9091d700 1 -- start start 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed8c071d20 0x7fed8c1f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c0d3a40 0x7fed8c15e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 0x7fed8c0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c15f200 con 0x7fed8c15e0c0 2024-01-16T13:53:55.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c15f4e0 con 0x7fed8c071d20 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c15f7c0 con 0x7fed8c0d3a40 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 0x7fed8c0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 0x7fed8c0c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47942/0 (socket says 172.21.15.113:47942) 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 -- 172.21.15.113:0/3056002256 learned_addr learned my addr 172.21.15.113:0/3056002256 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8a59c700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed8c071d20 0x7fed8c1f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed89d9b700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c0d3a40 0x7fed8c15e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 -- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c0d3a40 msgr2=0x7fed8c15e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c0d3a40 0x7fed8c15e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 -- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed8c071d20 msgr2=0x7fed8c1f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed8c071d20 0x7fed8c1f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 -- 172.21.15.113:0/3056002256 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84009a70 con 0x7fed8c15e0c0 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed89d9b700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c0d3a40 0x7fed8c15e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed8ad9d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 0x7fed8c0c9ab0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fed8400efb0 tx=0x7fed8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8400cd40 con 0x7fed8c15e0c0 2024-01-16T13:53:55.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8400f040 con 0x7fed8c15e0c0 2024-01-16T13:53:55.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8400cea0 con 0x7fed8c15e0c0 2024-01-16T13:53:55.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c0094a0 con 0x7fed8c15e0c0 2024-01-16T13:53:55.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.372+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c15e620 con 0x7fed8c15e0c0 2024-01-16T13:53:55.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.374+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c067f60 con 0x7fed8c15e0c0 2024-01-16T13:53:55.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.374+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fed840146a0 con 0x7fed8c15e0c0 2024-01-16T13:53:55.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.374+0000 7fed7affd700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fed6c06b500 0x7fed6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.374+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed8408adf0 con 0x7fed8c15e0c0 2024-01-16T13:53:55.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.374+0000 7fed8a59c700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fed6c06b500 0x7fed6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.375+0000 7fed8a59c700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fed6c06b500 0x7fed6c06d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:55.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.379+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed840557a0 con 0x7fed8c15e0c0 2024-01-16T13:53:55.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.606+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fed8c15f650 con 0x7fed6c06b500 2024-01-16T13:53:55.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.623+0000 7fed7affd700 1 -- 172.21.15.113:0/3056002256 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fed8c15f650 con 0x7fed6c06b500 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fed6c06b500 msgr2=0x7fed6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fed6c06b500 0x7fed6c06d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c008030 comp rx=0 tx=0).stop 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 msgr2=0x7fed8c0c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 0x7fed8c0c9ab0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fed8400efb0 tx=0x7fed8400c3a0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 shutdown_connections 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fed6c06b500 0x7fed6c06d9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed8c15e0c0 0x7fed8c0c9ab0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed8c071d20 0x7fed8c1f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 --2- 172.21.15.113:0/3056002256 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed8c0d3a40 0x7fed8c15e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 >> 172.21.15.113:0/3056002256 conn(0x7fed8c06b620 msgr2=0x7fed8c15c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:55.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 shutdown_connections 2024-01-16T13:53:55.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.626+0000 7fed9091d700 1 -- 172.21.15.113:0/3056002256 wait complete. 2024-01-16T13:53:55.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:53:55.621 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T13:53:55.656 INFO:teuthology.orchestra.run.smithi113.stdout:2 2024-01-16T13:53:55.657 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T13:53:55.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.839+0000 7fdc58b15700 1 Processor -- start 2024-01-16T13:53:55.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.839+0000 7fdc58b15700 1 -- start start 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc58b15700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc540cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc58b15700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5406d890 con 0x7fdc540afb00 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc58b15700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc540ccfb0 con 0x7fdc54071b30 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc58b15700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540cd290 con 0x7fdc5415bc80 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc51d9b700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fdc540afb00 0x7fdc54071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40744/0 (socket says 172.21.15.113:40744) 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc51d9b700 1 -- 172.21.15.113:0/1994856825 learned_addr learned my addr 172.21.15.113:0/1994856825 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5259c700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc540cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784433503 0 0) 0x7fdc5406d890 con 0x7fdc540afb00 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc3c0035b0 con 0x7fdc540afb00 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406928776 0 0) 0x7fdc540ccfb0 con 0x7fdc54071b30 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc5406d890 con 0x7fdc54071b30 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068236197 0 0) 0x7fdc3c0035b0 con 0x7fdc540afb00 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 msgr2=0x7fdc540cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc540cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.840+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 >> v1:172.21.15.124:6789/0 conn(0x7fdc54071b30 legacy=0x7fdc5406dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc540cd570 con 0x7fdc540afb00 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fdc44001670 con 0x7fdc540afb00 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5259c700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc540cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 >> v1:172.21.15.113:6789/0 conn(0x7fdc540afb00 legacy=0x7fdc54071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc3c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc3c004100 0x7fdc3c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc3c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540cd290 con 0x7fdc3c004d90 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c00bac0 con 0x7fdc3c004100 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c00bc80 con 0x7fdc5415bc80 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc3c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5259c700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc3c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc51d9b700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc3c004100 0x7fdc3c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 -- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 msgr2=0x7fdc3c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc5415bc80 0x7fdc3c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 -- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc3c004100 msgr2=0x7fdc3c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc3c004100 0x7fdc3c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 -- 172.21.15.113:0/1994856825 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480094a0 con 0x7fdc3c004d90 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc52d9d700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc3c005210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00c4c0 tx=0x7fdc3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3386c92a5de79df0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:55.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c003a00 con 0x7fdc3c004d90 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c003b60 con 0x7fdc3c004d90 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.841+0000 7fdc5159a700 1 -- 172.21.15.113:0/1994856825 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0050d0 con 0x7fdc3c004d90 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc58b15700 1 -- 172.21.15.113:0/1994856825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 msgr2=0x7fdc3c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc58b15700 1 --2- 172.21.15.113:0/1994856825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc3c005210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00c4c0 tx=0x7fdc3c005880 comp rx=0 tx=0).stop 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc52d9d700 1 -- 172.21.15.113:0/1994856825 reap_dead start 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc58b15700 1 -- 172.21.15.113:0/1994856825 shutdown_connections 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc58b15700 1 -- 172.21.15.113:0/1994856825 >> 172.21.15.113:0/1994856825 conn(0x7fdc54074170 msgr2=0x7fdc5406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc58b15700 1 -- 172.21.15.113:0/1994856825 shutdown_connections 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.842+0000 7fdc58b15700 1 -- 172.21.15.113:0/1994856825 wait complete. 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.843+0000 7fdc58b15700 1 Processor -- start 2024-01-16T13:53:55.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.843+0000 7fdc58b15700 1 -- start start 2024-01-16T13:53:55.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc58b15700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc54155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc58b15700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc54145b20 0x7fdc54153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc58b15700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc54154260 0x7fdc541f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc58b15700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54155e70 con 0x7fdc3c004d90 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc58b15700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54154890 con 0x7fdc54154260 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc58b15700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54154b70 con 0x7fdc54145b20 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc54155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc54155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47964/0 (socket says 172.21.15.113:47964) 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 -- 172.21.15.113:0/3646102448 learned_addr learned my addr 172.21.15.113:0/3646102448 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc51d9b700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc54145b20 0x7fdc54153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc52d9d700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc54154260 0x7fdc541f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 -- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc54145b20 msgr2=0x7fdc54153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc54145b20 0x7fdc54153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 -- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc54154260 msgr2=0x7fdc541f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:55.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc54154260 0x7fdc541f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.844+0000 7fdc5259c700 1 -- 172.21.15.113:0/3646102448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc54154e50 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.845+0000 7fdc5259c700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc54155930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdc48011cd0 tx=0x7fdc4800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.845+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48011e70 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.845+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4800fd40 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.845+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc54155130 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.845+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48014680 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.845+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc54150020 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.846+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5406bb30 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.846+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fdc480147e0 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.846+0000 7fdc42ffd700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc3406b500 0x7fdc3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.847+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc4808a8e0 con 0x7fdc3c004d90 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.847+0000 7fdc51d9b700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc3406b500 0x7fdc3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:55.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.847+0000 7fdc51d9b700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc3406b500 0x7fdc3406d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdc540cf690 tx=0x7fdc44009040 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:55.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:55.851+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc480554d0 con 0x7fdc3c004d90 2024-01-16T13:53:56.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:56.068+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 --> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc540cfaa0 con 0x7fdc3406b500 2024-01-16T13:53:56.078 INFO:tasks.ceph.mon.a.smithi113.stderr:2024-01-16T13:53:56.096+0000 7f9e78d0f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T13:53:56.138 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2024-01-16T13:53:58.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:58.124+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fdc480558a0 con 0x7fdc3c004d90 2024-01-16T13:53:59.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.133+0000 7fdc42ffd700 1 -- 172.21.15.113:0/3646102448 <== mgr.4113 v2:172.21.15.113:6832/1582984142 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fdc44007030 con 0x7fdc3406b500 2024-01-16T13:53:59.115 INFO:teuthology.orchestra.run.smithi113.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc3406b500 msgr2=0x7fdc3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc3406b500 0x7fdc3406d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdc540cf690 tx=0x7fdc44009040 comp rx=0 tx=0).stop 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 msgr2=0x7fdc54155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc54155930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdc48011cd0 tx=0x7fdc4800f520 comp rx=0 tx=0).stop 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 shutdown_connections 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fdc3406b500 0x7fdc3406d9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc54154260 0x7fdc541f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c004d90 0x7fdc54155930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 --2- 172.21.15.113:0/3646102448 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdc54145b20 0x7fdc54153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 >> 172.21.15.113:0/3646102448 conn(0x7fdc54074170 msgr2=0x7fdc5415f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:59.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 shutdown_connections 2024-01-16T13:53:59.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.137+0000 7fdc58b15700 1 -- 172.21.15.113:0/3646102448 wait complete. 2024-01-16T13:53:59.140 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T13:53:59.140 DEBUG:teuthology.orchestra.run.smithi113:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T13:53:59.198 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T13:53:59.295 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T13:53:59.296 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:53:59.296 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:59.314 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-01-16T13:53:59.314 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T13:53:59.371 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T13:53:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.550+0000 7fc5fda8a700 1 Processor -- start 2024-01-16T13:53:59.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.550+0000 7fc5fda8a700 1 -- start start 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5fda8a700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5f80b4350 0x7fc5f80af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5fda8a700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f80b4820 con 0x7fc5f80b59f0 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5fda8a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f80b4b00 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5fda8a700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f80ae9b0 con 0x7fc5f80b4350 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f7fff700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5f80b4350 0x7fc5f80af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f77fe700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc5f80b59f0 0x7fc5f806db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40746/0 (socket says 172.21.15.113:40746) 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f77fe700 1 -- 172.21.15.113:0/2051562172 learned_addr learned my addr 172.21.15.113:0/2051562172 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f7fff700 1 -- 172.21.15.113:0/2051562172 >> v1:172.21.15.124:6789/0 conn(0x7fc5f80b5dd0 legacy=0x7fc5f80b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f7fff700 1 -- 172.21.15.113:0/2051562172 >> v1:172.21.15.113:6789/0 conn(0x7fc5f80b59f0 legacy=0x7fc5f806db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f67fc700 1 -- 172.21.15.113:0/2051562172 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68726588 0 0) 0x7fc5f80b4820 con 0x7fc5f80b59f0 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f7fff700 1 -- 172.21.15.113:0/2051562172 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f80aec90 con 0x7fc5f80b4350 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f7fff700 1 --2- 172.21.15.113:0/2051562172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b4350 0x7fc5f80af190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0009dc0 tx=0x7fc5f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cfef33ff862ddd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f67fc700 1 -- 172.21.15.113:0/2051562172 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f000e070 con 0x7fc5f80b4350 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f67fc700 1 -- 172.21.15.113:0/2051562172 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f000bb90 con 0x7fc5f80b4350 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.551+0000 7fc5f67fc700 1 -- 172.21.15.113:0/2051562172 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0012430 con 0x7fc5f80b4350 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 -- 172.21.15.113:0/2051562172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b4350 msgr2=0x7fc5f80af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/2051562172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b4350 0x7fc5f80af190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0009dc0 tx=0x7fc5f000b330 comp rx=0 tx=0).stop 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 -- 172.21.15.113:0/2051562172 shutdown_connections 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/2051562172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b4350 0x7fc5f80af190 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 -- 172.21.15.113:0/2051562172 >> 172.21.15.113:0/2051562172 conn(0x7fc5f8072990 msgr2=0x7fc5f806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 -- 172.21.15.113:0/2051562172 shutdown_connections 2024-01-16T13:53:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.552+0000 7fc5fda8a700 1 -- 172.21.15.113:0/2051562172 wait complete. 2024-01-16T13:53:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.553+0000 7fc5fda8a700 1 Processor -- start 2024-01-16T13:53:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.553+0000 7fc5fda8a700 1 -- start start 2024-01-16T13:53:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5fda8a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f80b4350 0x7fc5f813ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5fda8a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5f80b59f0 0x7fc5f80ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5fda8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 0x7fc5f80cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5fda8a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f81522e0 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5fda8a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f814eb40 con 0x7fc5f80b4350 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5fda8a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8154490 con 0x7fc5f80b59f0 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 0x7fc5f80cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 0x7fc5f80cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47982/0 (socket says 172.21.15.113:47982) 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f7fff700 1 -- 172.21.15.113:0/427726272 learned_addr learned my addr 172.21.15.113:0/427726272 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f77fe700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f80b4350 0x7fc5f813ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f6ffd700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5f80b59f0 0x7fc5f80ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f7fff700 1 -- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5f80b59f0 msgr2=0x7fc5f80ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f7fff700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5f80b59f0 0x7fc5f80ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.554+0000 7fc5f7fff700 1 -- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f80b4350 msgr2=0x7fc5f813ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5f7fff700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f80b4350 0x7fc5f813ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5f7fff700 1 -- 172.21.15.113:0/427726272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f0009a70 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5f7fff700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 0x7fc5f80cd400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0014040 tx=0x7fc5f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f001b810 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f0020070 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f8154770 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f001bbb0 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.555+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f8154d30 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.556+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f8159590 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.556+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc5f001bd10 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.556+0000 7fc5effff700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc5d806b590 0x7fc5d806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.557+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc5f0022070 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.557+0000 7fc5f77fe700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc5d806b590 0x7fc5d806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.557+0000 7fc5f77fe700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc5d806b590 0x7fc5d806da50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8007b10 tx=0x7fc5e8007a60 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:59.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.561+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5f0056de0 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.752+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5f8067f60 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.753+0000 7fc5effff700 1 -- 172.21.15.113:0/427726272 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13067 (secure 0 0 0) 0x7fc5f005a400 con 0x7fc5f80b5dd0 2024-01-16T13:53:59.735 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-01-16T13:53:59.735 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":37,"fsid":"e77ab909-7680-4e53-a08d-a26ae42f1a8c","created":"2024-01-16T13:51:46.757570+0000","modified":"2024-01-16T13:53:59.127717+0000","last_up_change":"2024-01-16T13:51:56.663794+0000","last_in_change":"2024-01-16T13:51:49.318904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T13:51:58.444894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4429388e-5550-4ec2-a13b-35e14274a793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6817","nonce":1924924117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6819","nonce":1924924117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6823","nonce":1924924117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":1924924117},{"type":"v1","addr":"172.21.15.113:6821","nonce":1924924117}]},"public_addr":"172.21.15.113:6817/1924924117","cluster_addr":"172.21.15.113:6819/1924924117","heartbeat_back_addr":"172.21.15.113:6823/1924924117","heartbeat_front_addr":"172.21.15.113:6821/1924924117","state":["exists","up"]},{"osd":1,"uuid":"1f608d38-5fb0-450e-880c-5eaebd9d045a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6809","nonce":1956545854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6811","nonce":1956545854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6815","nonce":1956545854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1956545854},{"type":"v1","addr":"172.21.15.113:6813","nonce":1956545854}]},"public_addr":"172.21.15.113:6809/1956545854","cluster_addr":"172.21.15.113:6811/1956545854","heartbeat_back_addr":"172.21.15.113:6815/1956545854","heartbeat_front_addr":"172.21.15.113:6813/1956545854","state":["exists","up"]},{"osd":2,"uuid":"90bfcd09-fa8a-4659-a24e-fbfb5f73cd90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6825","nonce":1336088369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6827","nonce":1336088369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6831","nonce":1336088369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1336088369},{"type":"v1","addr":"172.21.15.113:6829","nonce":1336088369}]},"public_addr":"172.21.15.113:6825/1336088369","cluster_addr":"172.21.15.113:6827/1336088369","heartbeat_back_addr":"172.21.15.113:6831/1336088369","heartbeat_front_addr":"172.21.15.113:6829/1336088369","state":["exists","up"]},{"osd":3,"uuid":"391ffd90-1e6a-4e72-adc3-6ee6320a0913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6801","nonce":3595238614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6803","nonce":3595238614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6807","nonce":3595238614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":3595238614},{"type":"v1","addr":"172.21.15.113:6805","nonce":3595238614}]},"public_addr":"172.21.15.113:6801/3595238614","cluster_addr":"172.21.15.113:6803/3595238614","heartbeat_back_addr":"172.21.15.113:6807/3595238614","heartbeat_front_addr":"172.21.15.113:6805/3595238614","state":["exists","up"]},{"osd":4,"uuid":"4a6407f5-da2f-4594-8157-eb029642ee09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6801","nonce":2883525511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6803","nonce":2883525511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6807","nonce":2883525511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":2883525511},{"type":"v1","addr":"172.21.15.124:6805","nonce":2883525511}]},"public_addr":"172.21.15.124:6801/2883525511","cluster_addr":"172.21.15.124:6803/2883525511","heartbeat_back_addr":"172.21.15.124:6807/2883525511","heartbeat_front_addr":"172.21.15.124:6805/2883525511","state":["exists","up"]},{"osd":5,"uuid":"b802e6e8-2700-4a63-a29e-833fe899f1ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6825","nonce":23742019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6827","nonce":23742019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6831","nonce":23742019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":23742019},{"type":"v1","addr":"172.21.15.124:6829","nonce":23742019}]},"public_addr":"172.21.15.124:6825/23742019","cluster_addr":"172.21.15.124:6827/23742019","heartbeat_back_addr":"172.21.15.124:6831/23742019","heartbeat_front_addr":"172.21.15.124:6829/23742019","state":["exists","up"]},{"osd":6,"uuid":"a074ea89-c631-40d6-bb40-703de5636e78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6810","nonce":3167236260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6814","nonce":3167236260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6819","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6821","nonce":3167236260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6815","nonce":3167236260},{"type":"v1","addr":"172.21.15.124:6817","nonce":3167236260}]},"public_addr":"172.21.15.124:6810/3167236260","cluster_addr":"172.21.15.124:6814/3167236260","heartbeat_back_addr":"172.21.15.124:6821/3167236260","heartbeat_front_addr":"172.21.15.124:6817/3167236260","state":["exists","up"]},{"osd":7,"uuid":"7cda6133-ccbb-4129-ab9b-de80c3f615a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6809","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6811","nonce":2264364451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6813","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6816","nonce":2264364451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6823","nonce":2264364451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2264364451},{"type":"v1","addr":"172.21.15.124:6820","nonce":2264364451}]},"public_addr":"172.21.15.124:6811/2264364451","cluster_addr":"172.21.15.124:6816/2264364451","heartbeat_back_addr":"172.21.15.124:6823/2264364451","heartbeat_front_addr":"172.21.15.124:6820/2264364451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.496841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.454959+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.493489+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.200323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.462343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:56.615738+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.514730+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T13:51:54.489244+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.124:6835/1874427660":"2024-01-17T13:53:56.096352+0000","172.21.15.124:6833/1874427660":"2024-01-17T13:53:56.096352+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc5d806b590 msgr2=0x7fc5d806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc5d806b590 0x7fc5d806da50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8007b10 tx=0x7fc5e8007a60 comp rx=0 tx=0).stop 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 msgr2=0x7fc5f80cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 0x7fc5f80cd400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0014040 tx=0x7fc5f0004ad0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 shutdown_connections 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fc5d806b590 0x7fc5d806da50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5f80b5dd0 0x7fc5f80cd400 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f80b4350 0x7fc5f813ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 --2- 172.21.15.113:0/427726272 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5f80b59f0 0x7fc5f80ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 >> 172.21.15.113:0/427726272 conn(0x7fc5f8072990 msgr2=0x7fc5f806db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:59.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 shutdown_connections 2024-01-16T13:53:59.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.756+0000 7fc5fda8a700 1 -- 172.21.15.113:0/427726272 wait complete. 2024-01-16T13:53:59.750 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T13:51:58.444894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T13:53:59.750 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T13:53:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.892+0000 7fdeb0f47700 1 Processor -- start 2024-01-16T13:53:59.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.892+0000 7fdeb0f47700 1 -- start start 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeb0f47700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdeac15e1e0 0x7fdeac0d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeb0f47700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 0x7fdeac0d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeb0f47700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac0d30b0 con 0x7fdeac0b3050 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeb0f47700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac0ce730 con 0x7fdeac15e1e0 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeb0f47700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac162920 con 0x7fdeac0b2b00 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeaad9d700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fdeac0b3050 0x7fdeac0b5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40750/0 (socket says 172.21.15.113:40750) 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.893+0000 7fdeaad9d700 1 -- 172.21.15.113:0/12418226 learned_addr learned my addr 172.21.15.113:0/12418226 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/12418226 >> v2:172.21.15.124:3300/0 conn(0x7fdeac15e1e0 0x7fdeac0d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea9d9b700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 0x7fdeac0d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886770301 0 0) 0x7fdeac0d30b0 con 0x7fdeac0b3050 2024-01-16T13:53:59.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde940035b0 con 0x7fdeac0b3050 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 <== mon.0 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065429383 0 0) 0x7fde940035b0 con 0x7fdeac0b3050 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 msgr2=0x7fdeac0d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 0x7fdeac0d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 >> v2:172.21.15.124:3300/0 conn(0x7fdeac15e1e0 msgr2=0x7fdeac0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 --2- 172.21.15.113:0/12418226 >> v2:172.21.15.124:3300/0 conn(0x7fdeac15e1e0 0x7fdeac0d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0ce940 con 0x7fdeac0b3050 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 <== mon.0 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2083122169 0 0) 0x7fdea40019f0 con 0x7fdeac0b3050 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 >> v1:172.21.15.113:6789/0 conn(0x7fdeac0b3050 legacy=0x7fdeac0b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fde94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 0x7fde94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde94004d90 0x7fde94005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0ce730 con 0x7fde94004d90 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac162920 con 0x7fde94003a00 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9400ba10 con 0x7fdeac0b2b00 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fde94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaad9d700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde94004d90 0x7fde94005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdea9d9b700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 0x7fde94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 -- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 msgr2=0x7fde94004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac0b2b00 0x7fde94004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 -- 172.21.15.113:0/12418226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde94004d90 msgr2=0x7fde94005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde94004d90 0x7fde94005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaa59c700 1 -- 172.21.15.113:0/12418226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940035b0 con 0x7fde94003a00 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.894+0000 7fdeaad9d700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde94004d90 0x7fde94005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fde94003e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009b60 tx=0x7fde94005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be25d5f6536fccfd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00e6c0 con 0x7fde94003a00 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c00e820 con 0x7fde94003a00 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdea959a700 1 -- 172.21.15.113:0/12418226 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00eb90 con 0x7fde94003a00 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdeb0f47700 1 -- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 msgr2=0x7fde94003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/12418226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fde94003e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009b60 tx=0x7fde94005880 comp rx=0 tx=0).stop 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.895+0000 7fdeaad9d700 1 -- 172.21.15.113:0/12418226 reap_dead start 2024-01-16T13:53:59.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.896+0000 7fdeb0f47700 1 -- 172.21.15.113:0/12418226 shutdown_connections 2024-01-16T13:53:59.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.896+0000 7fdeb0f47700 1 -- 172.21.15.113:0/12418226 >> 172.21.15.113:0/12418226 conn(0x7fdeac0717f0 msgr2=0x7fdeac071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:53:59.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.896+0000 7fdeb0f47700 1 -- 172.21.15.113:0/12418226 shutdown_connections 2024-01-16T13:53:59.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.896+0000 7fdeb0f47700 1 -- 172.21.15.113:0/12418226 wait complete. 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.897+0000 7fdeb0f47700 1 Processor -- start 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.897+0000 7fdeb0f47700 1 -- start start 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeb0f47700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fdeac1f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeb0f47700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 0x7fdeac1602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeb0f47700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac1f6f40 0x7fdeac1f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeb0f47700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac1f7930 con 0x7fdeac1f34d0 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeb0f47700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac160bb0 con 0x7fde94003a00 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeb0f47700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac160e90 con 0x7fdeac1f6f40 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 0x7fdeac1602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 0x7fdeac1602a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48000/0 (socket says 172.21.15.113:48000) 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 -- 172.21.15.113:0/314842790 learned_addr learned my addr 172.21.15.113:0/314842790 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:53:59.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeaad9d700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac1f6f40 0x7fdeac1f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 -- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac1f6f40 msgr2=0x7fdeac1f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac1f6f40 0x7fdeac1f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 -- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 msgr2=0x7fdeac1f4560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fdeac1f4560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdea9d9b700 1 -- 172.21.15.113:0/314842790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac161170 con 0x7fdeac1f34d0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.899+0000 7fdeaad9d700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac1f6f40 0x7fdeac1f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.900+0000 7fdea9d9b700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 0x7fdeac1602a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdea0011aa0 tx=0x7fdea0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.900+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000f980 con 0x7fdeac1f34d0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.900+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea000fb00 con 0x7fdeac1f34d0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.900+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac161450 con 0x7fdeac1f34d0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.900+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000feb0 con 0x7fdeac1f34d0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.900+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac1607e0 con 0x7fdeac1f34d0 2024-01-16T13:53:59.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.901+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fdea0014440 con 0x7fdeac1f34d0 2024-01-16T13:53:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.902+0000 7fde9affd700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fde8c06b590 0x7fde8c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:53:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.902+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fdea008a4c0 con 0x7fdeac1f34d0 2024-01-16T13:53:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.902+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde900052f0 con 0x7fdeac1f34d0 2024-01-16T13:53:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.903+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fde8c06b590 0x7fde8c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:53:59.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.903+0000 7fdeaa59c700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fde8c06b590 0x7fde8c06da50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0d2db0 tx=0x7fde9c008030 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:53:59.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:53:59.913+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdea00553c0 con 0x7fdeac1f34d0 2024-01-16T13:54:00.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.158+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde90005b20 con 0x7fdeac1f34d0 2024-01-16T13:54:00.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.158+0000 7fde9affd700 1 -- 172.21.15.113:0/314842790 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdea0022070 con 0x7fdeac1f34d0 2024-01-16T13:54:00.140 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-01-16T13:54:00.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fde8c06b590 msgr2=0x7fde8c06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fde8c06b590 0x7fde8c06da50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0d2db0 tx=0x7fde9c008030 comp rx=0 tx=0).stop 2024-01-16T13:54:00.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 msgr2=0x7fdeac1602a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 0x7fdeac1602a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdea0011aa0 tx=0x7fdea0011e60 comp rx=0 tx=0).stop 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 shutdown_connections 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7fde8c06b590 0x7fde8c06da50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdeac1f6f40 0x7fdeac1f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fde94003a00 0x7fdeac1f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 --2- 172.21.15.113:0/314842790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdeac1f34d0 0x7fdeac1602a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.160+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 >> 172.21.15.113:0/314842790 conn(0x7fdeac0717f0 msgr2=0x7fdeac0ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.161+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 shutdown_connections 2024-01-16T13:54:00.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.161+0000 7fdeb0f47700 1 -- 172.21.15.113:0/314842790 wait complete. 2024-01-16T13:54:00.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T13:54:00.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.351+0000 7f2fe037d700 1 Processor -- start 2024-01-16T13:54:00.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.351+0000 7f2fe037d700 1 -- start start 2024-01-16T13:54:00.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.352+0000 7f2fe037d700 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2fd80d4570 0x7f2fd80b7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:54:00.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.352+0000 7f2fe037d700 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80ce6e0 con 0x7f2fd80d3720 2024-01-16T13:54:00.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.352+0000 7f2fe037d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80d3290 con 0x7f2fd81615d0 2024-01-16T13:54:00.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.352+0000 7f2fe037d700 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd80d0630 con 0x7f2fd80d4570 2024-01-16T13:54:00.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.352+0000 7f2fdd918700 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2fd80d3720 0x7f2fd80d3e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40756/0 (socket says 172.21.15.113:40756) 2024-01-16T13:54:00.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.352+0000 7f2fdd918700 1 -- 172.21.15.113:0/2468447110 learned_addr learned my addr 172.21.15.113:0/2468447110 (peer_addr_for_me v1:172.21.15.113:0/0) 2024-01-16T13:54:00.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fde91a700 1 --2- 172.21.15.113:0/2468447110 >> v2:172.21.15.113:3300/0 conn(0x7f2fd80d4570 0x7f2fd80b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:54:00.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fdd117700 1 -- 172.21.15.113:0/2468447110 <== mon.0 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389280894 0 0) 0x7f2fd80ce6e0 con 0x7f2fd80d3720 2024-01-16T13:54:00.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fdd117700 1 -- 172.21.15.113:0/2468447110 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc40035b0 con 0x7f2fd80d3720 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fde91a700 1 -- 172.21.15.113:0/2468447110 >> v1:172.21.15.124:6789/0 conn(0x7f2fd81615d0 legacy=0x7f2fd80b9860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fde91a700 1 -- 172.21.15.113:0/2468447110 >> v1:172.21.15.113:6789/0 conn(0x7f2fd80d3720 legacy=0x7f2fd80d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fde91a700 1 -- 172.21.15.113:0/2468447110 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80d3bd0 con 0x7f2fd80d4570 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fde91a700 1 --2- 172.21.15.113:0/2468447110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b7070 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009d90 tx=0x7f2fd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e24ac0f7aab2c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fdd117700 1 -- 172.21.15.113:0/2468447110 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd000e070 con 0x7f2fd80d4570 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fdd117700 1 -- 172.21.15.113:0/2468447110 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd000bae0 con 0x7f2fd80d4570 2024-01-16T13:54:00.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.353+0000 7f2fdd117700 1 -- 172.21.15.113:0/2468447110 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd000be90 con 0x7f2fd80d4570 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.354+0000 7f2fe037d700 1 -- 172.21.15.113:0/2468447110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 msgr2=0x7f2fd80b7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.354+0000 7f2fe037d700 1 --2- 172.21.15.113:0/2468447110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b7070 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009d90 tx=0x7f2fd000b330 comp rx=0 tx=0).stop 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.355+0000 7f2fe037d700 1 -- 172.21.15.113:0/2468447110 shutdown_connections 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.355+0000 7f2fe037d700 1 --2- 172.21.15.113:0/2468447110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b7070 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.355+0000 7f2fe037d700 1 -- 172.21.15.113:0/2468447110 >> 172.21.15.113:0/2468447110 conn(0x7f2fd806b410 msgr2=0x7f2fd806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.355+0000 7f2fe037d700 1 -- 172.21.15.113:0/2468447110 shutdown_connections 2024-01-16T13:54:00.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.355+0000 7f2fe037d700 1 -- 172.21.15.113:0/2468447110 wait complete. 2024-01-16T13:54:00.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.356+0000 7f2fe037d700 1 Processor -- start 2024-01-16T13:54:00.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.356+0000 7f2fe037d700 1 -- start start 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fe037d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2fd80d3720 0x7f2fd8150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fe037d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fe037d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fd81615d0 0x7f2fd80b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fe037d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80cc0d0 con 0x7f2fd80d4570 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fe037d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd815d910 con 0x7f2fd81615d0 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fe037d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80705d0 con 0x7f2fd80d3720 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fdd918700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fdd918700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48028/0 (socket says 172.21.15.113:48028) 2024-01-16T13:54:00.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.357+0000 7f2fdd918700 1 -- 172.21.15.113:0/1205361277 learned_addr learned my addr 172.21.15.113:0/1205361277 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.358+0000 7f2fdd918700 1 -- 172.21.15.113:0/1205361277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2fd80d3720 msgr2=0x7f2fd8150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.358+0000 7f2fdd918700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2fd80d3720 0x7f2fd8150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.358+0000 7f2fdd918700 1 -- 172.21.15.113:0/1205361277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fd81615d0 msgr2=0x7f2fd80b5260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.358+0000 7f2fdd918700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fd81615d0 0x7f2fd80b5260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.358+0000 7f2fdd918700 1 -- 172.21.15.113:0/1205361277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd0009a70 con 0x7f2fd80d4570 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.358+0000 7f2fdd918700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b4d20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c730 tx=0x7f2fd400caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:54:00.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.359+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4005d70 con 0x7f2fd80d4570 2024-01-16T13:54:00.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.359+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd40070d0 con 0x7f2fd80d4570 2024-01-16T13:54:00.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.359+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4007440 con 0x7f2fd80d4570 2024-01-16T13:54:00.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.359+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd80708b0 con 0x7f2fd80d4570 2024-01-16T13:54:00.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.359+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd8070e70 con 0x7f2fd80d4570 2024-01-16T13:54:00.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.360+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd8067f60 con 0x7f2fd80d4570 2024-01-16T13:54:00.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.361+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2fd40075a0 con 0x7f2fd80d4570 2024-01-16T13:54:00.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.362+0000 7f2fceffd700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2fbc06b5e0 0x7f2fbc06daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T13:54:00.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.362+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2fd40891e0 con 0x7f2fd80d4570 2024-01-16T13:54:00.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.362+0000 7f2fde119700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2fbc06b5e0 0x7f2fbc06daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T13:54:00.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.362+0000 7f2fde119700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2fbc06b5e0 0x7f2fbc06daa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80096d0 tx=0x7f2fc8006c30 comp rx=0 tx=0).ready entity=mgr.4113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T13:54:00.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.368+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fd4053fd0 con 0x7f2fd80d4570 2024-01-16T13:54:00.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.587+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2fd8071f70 con 0x7f2fd80d4570 2024-01-16T13:54:00.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.587+0000 7f2fceffd700 1 -- 172.21.15.113:0/1205361277 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v37) v1 ==== 80+0+124 (secure 0 0 0) 0x7f2fd40575f0 con 0x7f2fd80d4570 2024-01-16T13:54:00.570 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.124:6835/1874427660 2024-01-17T13:53:56.096352+0000 2024-01-16T13:54:00.570 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.124:6833/1874427660 2024-01-17T13:53:56.096352+0000 2024-01-16T13:54:00.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2fbc06b5e0 msgr2=0x7f2fbc06daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2fbc06b5e0 0x7f2fbc06daa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80096d0 tx=0x7f2fc8006c30 comp rx=0 tx=0).stop 2024-01-16T13:54:00.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 msgr2=0x7f2fd80b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T13:54:00.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b4d20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c730 tx=0x7f2fd400caf0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 shutdown_connections 2024-01-16T13:54:00.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:6832/1582984142,v1:172.21.15.113:6833/1582984142] conn(0x7f2fbc06b5e0 0x7f2fbc06daa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fd81615d0 0x7f2fd80b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2fd80d3720 0x7f2fd8150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 --2- 172.21.15.113:0/1205361277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fd80d4570 0x7f2fd80b4d20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 >> 172.21.15.113:0/1205361277 conn(0x7f2fd806b410 msgr2=0x7f2fd80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 shutdown_connections 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-01-16T13:54:00.590+0000 7f2fe037d700 1 -- 172.21.15.113:0/1205361277 wait complete. 2024-01-16T13:54:00.573 INFO:teuthology.orchestra.run.smithi113.stderr:listed 2 entries 2024-01-16T13:54:00.584 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-01-16T13:54:00.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T13:54:00.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-01-16T13:54:10.675 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:54:10.678 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:54:40.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:54:40.758 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:55:10.834 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:55:10.837 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:55:40.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:55:40.917 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:56:11.007 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:56:11.010 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:56:41.113 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:56:41.116 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:57:11.185 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:57:11.188 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:57:41.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:57:41.253 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:58:11.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:58:11.325 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:58:41.388 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:58:41.391 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:59:00.587 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-01-16T13:59:00.637 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-01-16T13:59:11.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:59:11.478 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:59:41.552 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T13:59:41.555 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:00:11.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:00:11.637 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:00:41.701 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:00:41.704 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:01:11.773 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:01:11.777 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:01:41.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:01:41.850 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:02:11.993 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:02:11.996 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:02:42.053 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:02:42.057 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:03:12.115 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:03:12.118 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:03:42.219 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:03:42.223 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:04:12.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:04:12.292 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:04:42.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:04:42.362 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:05:12.430 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:05:12.434 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:05:42.521 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:05:42.524 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:06:12.588 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:06:12.591 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:06:42.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:06:42.665 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:07:12.762 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:07:12.765 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:07:42.833 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:07:42.841 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:08:12.914 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:08:12.918 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:08:43.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:08:43.025 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:09:13.094 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:09:13.098 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:09:43.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:09:43.169 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:10:13.311 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:10:13.315 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:10:43.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:10:43.401 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:11:13.468 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:11:13.471 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:11:43.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:11:43.542 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:12:13.622 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:12:13.625 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:12:43.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:12:43.701 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:13:13.793 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:13:13.796 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:13:43.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:13:43.871 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:14:13.943 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:14:13.946 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:14:44.069 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:14:44.073 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:15:14.189 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:15:14.192 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:15:44.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:15:44.295 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:16:14.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:16:14.424 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:16:44.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:16:44.495 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:17:14.590 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:17:14.595 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:17:44.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:17:44.734 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:18:14.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:18:14.871 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:18:45.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:18:45.009 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:19:15.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:19:15.112 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:19:45.250 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:19:45.254 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:20:15.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:20:15.399 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:20:45.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:20:45.471 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:21:15.541 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:21:15.544 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:21:45.690 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:21:45.694 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:22:15.836 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:22:15.842 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:22:45.965 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:22:45.968 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:23:16.102 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:23:16.104 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:23:46.247 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:23:46.251 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:24:16.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:24:16.362 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:24:46.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:24:46.500 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:25:16.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:25:16.571 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:25:46.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:25:46.639 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:26:16.765 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:26:16.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:26:46.903 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:26:46.906 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:27:17.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:27:17.068 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:27:47.203 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:27:47.207 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:28:17.264 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:28:17.267 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:28:47.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:28:47.366 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:29:17.444 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:29:17.448 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:29:47.585 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:29:47.588 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:30:17.714 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:30:17.717 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:30:48.078 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:30:48.082 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:31:18.156 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:31:18.159 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:31:48.286 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:31:48.290 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:32:18.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:32:18.375 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:32:48.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:32:48.512 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:33:18.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:33:18.576 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:33:48.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:33:48.702 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:34:18.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:34:18.778 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:34:48.849 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:34:48.853 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:35:18.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:35:18.925 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:35:49.021 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:35:49.023 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:36:19.094 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:36:19.097 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:36:49.167 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:36:49.172 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:37:19.249 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:37:19.254 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:37:49.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:37:49.324 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:38:19.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:38:19.397 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:38:25.710 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:38:25.711 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.0... 2024-01-16T14:38:25.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:38:25.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:38:25.714 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:38:28.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-16T14:38:28.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-16T14:38:28.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-16T14:38:28.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-16T14:38:33.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-01-16T14:38:33.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-01-16T14:38:33.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~5s 2024-01-16T14:38:33.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2024-01-16T14:38:34.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:38:34.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:38:38.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~10s 2024-01-16T14:38:38.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~10s 2024-01-16T14:38:38.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~10s 2024-01-16T14:38:38.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~10s 2024-01-16T14:38:38.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~10s 2024-01-16T14:38:38.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~10s 2024-01-16T14:38:38.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~10s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~15s 2024-01-16T14:38:43.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~15s 2024-01-16T14:38:43.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~15s 2024-01-16T14:38:48.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~20s 2024-01-16T14:38:48.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~20s 2024-01-16T14:38:48.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~20s 2024-01-16T14:38:48.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~20s 2024-01-16T14:38:49.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:38:49.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:38:53.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~25s 2024-01-16T14:38:53.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~25s 2024-01-16T14:38:53.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~25s 2024-01-16T14:38:53.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~25s 2024-01-16T14:38:58.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~30s 2024-01-16T14:38:58.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~30s 2024-01-16T14:38:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~30s 2024-01-16T14:39:03.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~35s 2024-01-16T14:39:03.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~35s 2024-01-16T14:39:03.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~35s 2024-01-16T14:39:03.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~35s 2024-01-16T14:39:03.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~35s 2024-01-16T14:39:03.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~35s 2024-01-16T14:39:07.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:07.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:08.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~40s 2024-01-16T14:39:08.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~40s 2024-01-16T14:39:13.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~45s 2024-01-16T14:39:13.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~45s 2024-01-16T14:39:13.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~45s 2024-01-16T14:39:16.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:16.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:18.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~50s 2024-01-16T14:39:18.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2024-01-16T14:39:18.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~50s 2024-01-16T14:39:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~50s 2024-01-16T14:39:19.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:19.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:19.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:19.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:22.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:22.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:22.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:22.972 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ERROR 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner:ERROR: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-01-16T14:39:22.999 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 160, in umount 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 148, in wait 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-01-16T14:39:23.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: for line in f: 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner:socket.timeout 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7609, in tearDown 2024-01-16T14:39:23.001 INFO:tasks.cephfs_test_runner: super(self.__class__, self).tearDown() 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 459, in tearDown 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/cephfs_test_case.py", line 205, in tearDown 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: m.teardown() 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 207, in teardown 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: self.umount() 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 163, in umount 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: self.client_remote.run( 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner:ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner:Ran 1 test in 2809.524s 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-01-16T14:39:23.002 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:ERROR: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 160, in umount 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T14:39:23.003 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 148, in wait 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: for line in f: 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-01-16T14:39:23.004 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner:socket.timeout 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7609, in tearDown 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: super(self.__class__, self).tearDown() 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 459, in tearDown 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: super(TestVolumesHelper, self).tearDown() 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/cephfs_test_case.py", line 205, in tearDown 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: m.teardown() 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 207, in teardown 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: self.umount() 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 163, in umount 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: self.client_remote.run( 2024-01-16T14:39:23.005 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run 2024-01-16T14:39:23.006 INFO:tasks.cephfs_test_runner: raise ConnectionError(f'Failed to reconnect to {self.shortname}') 2024-01-16T14:39:23.006 INFO:tasks.cephfs_test_runner:ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:39:23.006 INFO:tasks.cephfs_test_runner: 2024-01-16T14:39:23.006 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:39:23.334 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=51c857c4082d433d845a80bb63f5e8fb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T14:39:23.335 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-16T14:39:23.344 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-01-16T14:39:23.377 INFO:tasks.kclient:Unmounting kernel clients... 2024-01-16T14:39:23.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:23.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~55s 2024-01-16T14:39:23.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~55s 2024-01-16T14:39:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~55s 2024-01-16T14:39:23.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~55s 2024-01-16T14:39:26.432 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:28.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~60s 2024-01-16T14:39:28.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~60s 2024-01-16T14:39:28.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~60s 2024-01-16T14:39:28.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~60s 2024-01-16T14:39:28.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~60s 2024-01-16T14:39:28.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~60s 2024-01-16T14:39:28.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~60s 2024-01-16T14:39:28.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~60s 2024-01-16T14:39:31.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:31.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:33.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~65s 2024-01-16T14:39:33.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-01-16T14:39:33.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~65s 2024-01-16T14:39:33.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~65s 2024-01-16T14:39:33.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~65s 2024-01-16T14:39:33.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~65s 2024-01-16T14:39:33.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~65s 2024-01-16T14:39:33.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~65s 2024-01-16T14:39:33.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~65s 2024-01-16T14:39:34.975 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:35.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:35.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:37.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:37.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:38.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:38.045 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:39:38.045 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 184, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:39:38.051 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:38.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~70s 2024-01-16T14:39:38.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~70s 2024-01-16T14:39:38.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~70s 2024-01-16T14:39:43.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~75s 2024-01-16T14:39:43.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~75s 2024-01-16T14:39:43.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~75s 2024-01-16T14:39:43.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~75s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~80s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~80s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~80s 2024-01-16T14:39:48.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~80s 2024-01-16T14:39:48.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~80s 2024-01-16T14:39:48.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~80s 2024-01-16T14:39:50.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:50.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:53.120 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:53.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-01-16T14:39:53.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2024-01-16T14:39:53.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~85s 2024-01-16T14:39:53.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~85s 2024-01-16T14:39:53.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~85s 2024-01-16T14:39:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~85s 2024-01-16T14:39:53.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~85s 2024-01-16T14:39:53.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~85s 2024-01-16T14:39:53.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~85s 2024-01-16T14:39:56.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:56.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:56.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:39:56.188 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:39:56.188 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:39:56.195 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T14:39:56.207 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T14:39:56.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:39:56.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:39:58.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2024-01-16T14:39:58.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2024-01-16T14:39:58.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2024-01-16T14:39:58.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~90s 2024-01-16T14:39:58.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~90s 2024-01-16T14:39:58.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~90s 2024-01-16T14:39:58.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~90s 2024-01-16T14:39:58.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~90s 2024-01-16T14:39:58.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~90s 2024-01-16T14:39:58.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~90s 2024-01-16T14:39:59.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:03.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-01-16T14:40:03.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2024-01-16T14:40:03.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~95s 2024-01-16T14:40:03.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~95s 2024-01-16T14:40:03.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~95s 2024-01-16T14:40:03.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~95s 2024-01-16T14:40:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~95s 2024-01-16T14:40:03.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~95s 2024-01-16T14:40:03.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~95s 2024-01-16T14:40:03.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~95s 2024-01-16T14:40:08.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:08.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:40:08.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~100s 2024-01-16T14:40:08.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~100s 2024-01-16T14:40:08.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~100s 2024-01-16T14:40:08.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~100s 2024-01-16T14:40:08.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~100s 2024-01-16T14:40:08.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~100s 2024-01-16T14:40:08.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~100s 2024-01-16T14:40:08.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~100s 2024-01-16T14:40:08.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~100s 2024-01-16T14:40:11.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:13.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~105s 2024-01-16T14:40:13.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~105s 2024-01-16T14:40:13.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~105s 2024-01-16T14:40:13.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~105s 2024-01-16T14:40:13.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~105s 2024-01-16T14:40:13.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~105s 2024-01-16T14:40:13.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~105s 2024-01-16T14:40:13.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~105s 2024-01-16T14:40:13.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~105s 2024-01-16T14:40:18.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~110s 2024-01-16T14:40:18.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2024-01-16T14:40:18.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~110s 2024-01-16T14:40:18.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~110s 2024-01-16T14:40:18.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~110s 2024-01-16T14:40:18.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~110s 2024-01-16T14:40:18.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~110s 2024-01-16T14:40:18.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~110s 2024-01-16T14:40:18.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~110s 2024-01-16T14:40:23.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:23.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:40:23.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~115s 2024-01-16T14:40:23.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~115s 2024-01-16T14:40:23.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~115s 2024-01-16T14:40:23.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~115s 2024-01-16T14:40:23.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~115s 2024-01-16T14:40:23.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~115s 2024-01-16T14:40:23.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~115s 2024-01-16T14:40:23.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~115s 2024-01-16T14:40:23.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~115s 2024-01-16T14:40:26.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:28.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~120s 2024-01-16T14:40:28.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~120s 2024-01-16T14:40:28.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~120s 2024-01-16T14:40:28.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~120s 2024-01-16T14:40:28.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~120s 2024-01-16T14:40:28.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~120s 2024-01-16T14:40:28.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~120s 2024-01-16T14:40:28.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~120s 2024-01-16T14:40:28.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~120s 2024-01-16T14:40:28.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~120s 2024-01-16T14:40:29.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:29.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:40:29.468 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:29.468 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:40:29.468 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:40:29.483 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T14:40:29.483 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T14:40:29.483 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.486 ERROR:teuthology.misc:Saw exception from ceph.mds.a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-01-16T14:40:29.500 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T14:40:29.500 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.501 ERROR:teuthology.misc:Saw exception from ceph.mds.c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-01-16T14:40:29.515 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T14:40:29.515 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.515 ERROR:teuthology.misc:Saw exception from ceph.mds.b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2024-01-16T14:40:29.529 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T14:40:29.529 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.529 ERROR:teuthology.misc:Saw exception from ceph.mds.d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2024-01-16T14:40:29.543 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T14:40:29.543 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T14:40:29.543 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.544 ERROR:teuthology.misc:Saw exception from ceph.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0' 2024-01-16T14:40:29.561 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T14:40:29.561 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.561 ERROR:teuthology.misc:Saw exception from ceph.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1' 2024-01-16T14:40:29.579 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T14:40:29.579 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.580 ERROR:teuthology.misc:Saw exception from ceph.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2' 2024-01-16T14:40:29.596 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-16T14:40:29.596 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.597 ERROR:teuthology.misc:Saw exception from ceph.osd.3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3' 2024-01-16T14:40:29.614 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-16T14:40:29.614 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.615 ERROR:teuthology.misc:Saw exception from ceph.osd.4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4' 2024-01-16T14:40:29.632 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-16T14:40:29.632 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.632 ERROR:teuthology.misc:Saw exception from ceph.osd.5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5' 2024-01-16T14:40:29.649 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-16T14:40:29.649 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.649 ERROR:teuthology.misc:Saw exception from ceph.osd.6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6' 2024-01-16T14:40:29.666 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-16T14:40:29.666 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.666 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.666 ERROR:teuthology.misc:Saw exception from ceph.osd.7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2024-01-16T14:40:29.682 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T14:40:29.682 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-16T14:40:29.682 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.683 ERROR:teuthology.misc:Saw exception from ceph.mgr.y Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-01-16T14:40:29.703 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T14:40:29.703 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.703 ERROR:teuthology.misc:Saw exception from ceph.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2024-01-16T14:40:29.723 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T14:40:29.723 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T14:40:29.723 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.723 ERROR:teuthology.misc:Saw exception from ceph.mon.a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi113 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a' 2024-01-16T14:40:29.748 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-16T14:40:29.748 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.748 ERROR:teuthology.misc:Saw exception from ceph.mon.b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b' 2024-01-16T14:40:29.774 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-16T14:40:29.774 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T14:40:29.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T14:40:29.774 ERROR:teuthology.misc:Saw exception from ceph.mon.c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1180, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi124 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c' 2024-01-16T14:40:29.798 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T14:40:29.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:29.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:40:32.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:33.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~125s 2024-01-16T14:40:33.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2024-01-16T14:40:33.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~125s 2024-01-16T14:40:33.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~125s 2024-01-16T14:40:33.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~125s 2024-01-16T14:40:33.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~125s 2024-01-16T14:40:33.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~125s 2024-01-16T14:40:33.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~125s 2024-01-16T14:40:33.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~125s 2024-01-16T14:40:33.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~125s 2024-01-16T14:40:33.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~125s 2024-01-16T14:40:38.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2024-01-16T14:40:38.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~130s 2024-01-16T14:40:38.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~130s 2024-01-16T14:40:38.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~130s 2024-01-16T14:40:38.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~130s 2024-01-16T14:40:41.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:41.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:40:43.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2024-01-16T14:40:43.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-01-16T14:40:43.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~135s 2024-01-16T14:40:43.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~135s 2024-01-16T14:40:43.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~135s 2024-01-16T14:40:44.604 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:48.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~140s 2024-01-16T14:40:48.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~140s 2024-01-16T14:40:48.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~140s 2024-01-16T14:40:53.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~145s 2024-01-16T14:40:53.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~145s 2024-01-16T14:40:53.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~145s 2024-01-16T14:40:53.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~145s 2024-01-16T14:40:56.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:56.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:40:56.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~150s 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~150s 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~150s 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~150s 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~150s 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~150s 2024-01-16T14:40:58.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~150s 2024-01-16T14:40:58.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~150s 2024-01-16T14:40:58.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~150s 2024-01-16T14:40:59.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:40:59.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:02.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:02.716 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:41:02.717 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T14:41:02.719 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127/remote/smithi113/crash 2024-01-16T14:41:02.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:41:02.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:03.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~155s 2024-01-16T14:41:03.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~155s 2024-01-16T14:41:03.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~155s 2024-01-16T14:41:03.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~155s 2024-01-16T14:41:03.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~155s 2024-01-16T14:41:03.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~155s 2024-01-16T14:41:03.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~155s 2024-01-16T14:41:03.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~155s 2024-01-16T14:41:03.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~155s 2024-01-16T14:41:03.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~155s 2024-01-16T14:41:05.788 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:08.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2024-01-16T14:41:08.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2024-01-16T14:41:08.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2024-01-16T14:41:08.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2024-01-16T14:41:08.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~160s 2024-01-16T14:41:08.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2024-01-16T14:41:08.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~160s 2024-01-16T14:41:08.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~160s 2024-01-16T14:41:08.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~160s 2024-01-16T14:41:08.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~160s 2024-01-16T14:41:08.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~160s 2024-01-16T14:41:13.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~165s 2024-01-16T14:41:13.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~165s 2024-01-16T14:41:13.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~165s 2024-01-16T14:41:14.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:41:14.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:17.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:18.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-01-16T14:41:18.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2024-01-16T14:41:18.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~170s 2024-01-16T14:41:18.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2024-01-16T14:41:18.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~170s 2024-01-16T14:41:18.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~170s 2024-01-16T14:41:18.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~170s 2024-01-16T14:41:18.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~170s 2024-01-16T14:41:18.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~170s 2024-01-16T14:41:18.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~170s 2024-01-16T14:41:23.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2024-01-16T14:41:23.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2024-01-16T14:41:23.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~175s 2024-01-16T14:41:23.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~175s 2024-01-16T14:41:23.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~175s 2024-01-16T14:41:23.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~175s 2024-01-16T14:41:23.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~175s 2024-01-16T14:41:23.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~175s 2024-01-16T14:41:23.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~175s 2024-01-16T14:41:23.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~175s 2024-01-16T14:41:28.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~180s 2024-01-16T14:41:28.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~180s 2024-01-16T14:41:28.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~180s 2024-01-16T14:41:29.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:41:29.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:29.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:32.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:41:32.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:33.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~185s 2024-01-16T14:41:33.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~185s 2024-01-16T14:41:33.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~185s 2024-01-16T14:41:35.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:35.964 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:41:35.965 INFO:tasks.ceph:Shutting down logrotate 2024-01-16T14:41:35.966 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 257, in ceph_log logrotater.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 222, in end self.thread.get() File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph.py", line 184, in invoke_logrotate procs = ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:41:35.989 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T14:41:35.998 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:41:36.012 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T14:41:36.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:41:36.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:38.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~190s 2024-01-16T14:41:38.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~190s 2024-01-16T14:41:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~190s 2024-01-16T14:41:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~190s 2024-01-16T14:41:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~190s 2024-01-16T14:41:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~190s 2024-01-16T14:41:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~190s 2024-01-16T14:41:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~190s 2024-01-16T14:41:38.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~190s 2024-01-16T14:41:39.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:43.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~195s 2024-01-16T14:41:43.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2024-01-16T14:41:43.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2024-01-16T14:41:43.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~195s 2024-01-16T14:41:43.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2024-01-16T14:41:43.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~195s 2024-01-16T14:41:43.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~195s 2024-01-16T14:41:43.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~195s 2024-01-16T14:41:43.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~195s 2024-01-16T14:41:43.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~195s 2024-01-16T14:41:43.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~195s 2024-01-16T14:41:48.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:41:48.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:41:48.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~200s 2024-01-16T14:41:48.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~200s 2024-01-16T14:41:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~200s 2024-01-16T14:41:51.100 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:41:53.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~205s 2024-01-16T14:41:53.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~205s 2024-01-16T14:41:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~205s 2024-01-16T14:41:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~210s 2024-01-16T14:41:58.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~210s 2024-01-16T14:41:58.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~210s 2024-01-16T14:41:58.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~210s 2024-01-16T14:42:03.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:03.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:03.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:03.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~215s 2024-01-16T14:42:03.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~215s 2024-01-16T14:42:03.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~215s 2024-01-16T14:42:06.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:06.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:08.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2024-01-16T14:42:08.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2024-01-16T14:42:08.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~220s 2024-01-16T14:42:08.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~220s 2024-01-16T14:42:08.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~220s 2024-01-16T14:42:08.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~220s 2024-01-16T14:42:08.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~220s 2024-01-16T14:42:08.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~220s 2024-01-16T14:42:08.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~220s 2024-01-16T14:42:09.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:09.212 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:42:09.265 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:42:09.280 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T14:42:09.290 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T14:42:09.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:09.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:12.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:13.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~225s 2024-01-16T14:42:13.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~225s 2024-01-16T14:42:13.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~225s 2024-01-16T14:42:13.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~225s 2024-01-16T14:42:18.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~231s 2024-01-16T14:42:18.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~231s 2024-01-16T14:42:18.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~231s 2024-01-16T14:42:18.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~231s 2024-01-16T14:42:18.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~231s 2024-01-16T14:42:18.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~231s 2024-01-16T14:42:18.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~231s 2024-01-16T14:42:18.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~231s 2024-01-16T14:42:18.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~231s 2024-01-16T14:42:21.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:21.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:23.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~236s 2024-01-16T14:42:23.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~236s 2024-01-16T14:42:23.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~236s 2024-01-16T14:42:23.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~236s 2024-01-16T14:42:23.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~236s 2024-01-16T14:42:23.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~236s 2024-01-16T14:42:23.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~236s 2024-01-16T14:42:23.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~236s 2024-01-16T14:42:24.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:28.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~241s 2024-01-16T14:42:28.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~241s 2024-01-16T14:42:28.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~241s 2024-01-16T14:42:28.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~241s 2024-01-16T14:42:33.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~246s 2024-01-16T14:42:33.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~246s 2024-01-16T14:42:33.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~246s 2024-01-16T14:42:36.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:36.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:36.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:38.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~251s 2024-01-16T14:42:38.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~251s 2024-01-16T14:42:38.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~251s 2024-01-16T14:42:38.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~251s 2024-01-16T14:42:39.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:39.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:42.465 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:42.465 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:42:42.466 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:42:42.480 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T14:42:42.491 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T14:42:42.492 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T14:42:42.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:42.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:43.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~256s 2024-01-16T14:42:43.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~256s 2024-01-16T14:42:43.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~256s 2024-01-16T14:42:43.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~256s 2024-01-16T14:42:43.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~256s 2024-01-16T14:42:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~256s 2024-01-16T14:42:43.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~256s 2024-01-16T14:42:43.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~256s 2024-01-16T14:42:45.536 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:48.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~261s 2024-01-16T14:42:48.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~261s 2024-01-16T14:42:48.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~261s 2024-01-16T14:42:53.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~266s 2024-01-16T14:42:53.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~266s 2024-01-16T14:42:53.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~266s 2024-01-16T14:42:53.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~266s 2024-01-16T14:42:53.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~266s 2024-01-16T14:42:53.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~266s 2024-01-16T14:42:53.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~266s 2024-01-16T14:42:53.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~266s 2024-01-16T14:42:53.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~266s 2024-01-16T14:42:54.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:42:54.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:42:57.596 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:42:58.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~271s 2024-01-16T14:42:58.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~271s 2024-01-16T14:42:58.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~271s 2024-01-16T14:42:58.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~271s 2024-01-16T14:42:58.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~271s 2024-01-16T14:42:58.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~271s 2024-01-16T14:42:58.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~271s 2024-01-16T14:42:58.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~271s 2024-01-16T14:42:58.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~271s 2024-01-16T14:43:03.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2024-01-16T14:43:03.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~276s 2024-01-16T14:43:03.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~276s 2024-01-16T14:43:03.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~276s 2024-01-16T14:43:03.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~276s 2024-01-16T14:43:03.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~276s 2024-01-16T14:43:03.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~276s 2024-01-16T14:43:03.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~276s 2024-01-16T14:43:03.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~276s 2024-01-16T14:43:08.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~281s 2024-01-16T14:43:08.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~281s 2024-01-16T14:43:08.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~281s 2024-01-16T14:43:09.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:09.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:09.724 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:12.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:12.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:13.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~286s 2024-01-16T14:43:13.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~286s 2024-01-16T14:43:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~286s 2024-01-16T14:43:13.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~286s 2024-01-16T14:43:15.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:15.708 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:43:15.709 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:43:15.718 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T14:43:15.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T14:43:15.774 INFO:teuthology.task.internal:Duration was 3723.250648 seconds 2024-01-16T14:43:15.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T14:43:15.816 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T14:43:15.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:15.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:18.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~291s 2024-01-16T14:43:18.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~291s 2024-01-16T14:43:18.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~291s 2024-01-16T14:43:18.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~291s 2024-01-16T14:43:18.784 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:23.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~296s 2024-01-16T14:43:23.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~296s 2024-01-16T14:43:23.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~296s 2024-01-16T14:43:23.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~296s 2024-01-16T14:43:23.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~296s 2024-01-16T14:43:23.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~296s 2024-01-16T14:43:23.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~296s 2024-01-16T14:43:23.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~296s 2024-01-16T14:43:23.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~296s 2024-01-16T14:43:27.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:27.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:28.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~301s 2024-01-16T14:43:28.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~301s 2024-01-16T14:43:28.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~301s 2024-01-16T14:43:28.671 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-01-16T14:43:28.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:28.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:30.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:30.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:39.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:39.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:39.900 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:42.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:42.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:42.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:45.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:45.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:48.989 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:48.989 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:43:48.990 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:43:49.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T14:43:49.023 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T14:43:49.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:49.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:51.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:51.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:52.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:52.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:55.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:55.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:55.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:43:55.132 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:43:55.132 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:43:55.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:43:55.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:43:58.209 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:01.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:01.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:01.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:07.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:07.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:10.268 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:13.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:13.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:13.340 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:16.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:16.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:19.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:19.324 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:44:19.325 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:44:19.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T14:44:19.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:19.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:22.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:22.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:22.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:22.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:25.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:25.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:25.472 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:25.472 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:44:25.472 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:44:25.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:44:25.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:26.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:44:31.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:31.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:34.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:35.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:44:35.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:35.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:44:46.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:46.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:46.588 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:47.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:44:47.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:47.932 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:44:49.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:49.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:50.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:44:50.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:52.608 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:52.608 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:44:52.609 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:44:52.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T14:44:52.630 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T14:44:52.631 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518127/remote/smithi113 2024-01-16T14:44:52.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:44:52.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:53.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:44:53.916 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:44:53.917 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi124 2024-01-16T14:44:53.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:44:53.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:55.680 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:44:56.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:45:04.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:04.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:05.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:45:05.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:07.776 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:09.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:45:19.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:19.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:19.932 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:21.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:45:21.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:21.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:45:22.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:22.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:24.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2024-01-16T14:45:24.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:25.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:25.917 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:45:25.918 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:45:25.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T14:45:25.948 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T14:45:25.948 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T14:45:25.977 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T14:45:25.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:25.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:27.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2024-01-16T14:45:27.168 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:45:27.168 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/kernel_mount.py", line 183, in umount_wait if not self.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi124 2024-01-16T14:45:28.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:37.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:37.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:41.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:53.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:53.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:53.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:56.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-01-16T14:45:56.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:45:59.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-01-16T14:45:59.164 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-16T14:45:59.165 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:45:59.176 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T14:45:59.190 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T14:45:59.226 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionError: Failed to reconnect to smithi113 2024-01-16T14:45:59.226 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T14:45:59.255 INFO:teuthology.nuke:Not nuking smithi113.front.sepia.ceph.com because description doesn't match: rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi != None 2024-01-16T14:45:59.280 INFO:teuthology.nuke:Not nuking smithi124.front.sepia.ceph.com because description doesn't match: rishabh-2024-01-16_13:34:04-fs:volumes-rishabh-clone-stats-testing-default-smithi != None 2024-01-16T14:45:59.281 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMgWGcqZsqMrhLUrN+rzIKJxjgom5ezT+W4YHAhRRit0V/oXcrcFOoaNVAfVwWQB1r4CmW4qRPPv3PnYCrSBGEE= smithi124.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGv3mk9zPlpUpN698e9yQhF8X8unVOZl5D0AVavTlbH00i8d2sN7KnsA82RC5i+0EbqLMIENWV7ImBDF2RWbi4Q= 2024-01-16T14:45:59.288 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 3723.250647544861 failure_reason: 'Test failure: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=51c857c4082d433d845a80bb63f5e8fb status: fail success: false 2024-01-16T14:45:59.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T14:45:59.369 INFO:teuthology.run:FAIL